Skip to content

Coherence Metrics

NPMI

Bases: BaseMetric

Normalized Pointwise Mutual Information (NPMI) coherence metric for topic modeling evaluation.

This class calculates the NPMI coherence score, which measures the coherence of topics based on the normalized pointwise mutual information of word pairs.

Key Features
  • Measures topic coherence using normalized pointwise mutual information
  • Suitable for evaluating topic models
  • Handles input data as pandas DataFrames
Usage

The NPMI metric can be used to evaluate topic modeling results:

from framework3.plugins.metrics.coherence import NPMI
from framework3.base.base_types import XYData
import pandas as pd
import numpy as np

# Assuming you have a DataFrame 'df' with your document-term matrix
x_data = XYData(value=df)
y_pred = np.array([['word1', 'word2', 'word3'], ['word4', 'word5', 'word6']])  # Example topics

npmi_metric = NPMI()
score = npmi_metric.evaluate(x_data, None, y_pred, f_vocab=df.columns)
print(f"NPMI Score: {score}")

Methods:

Name Description
evaluate

XYData, y_true: XYData | None, y_pred: XYData, **kwargs) -> Float | np.ndarray: Calculate the NPMI coherence score.
Note

This metric requires the input data to be a pandas DataFrame. Ensure that your data is properly formatted before using this metric.

Source code in framework3/plugins/metrics/coherence.py 
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
@Container.bind()
class NPMI(BaseMetric):
    """
    Normalized Pointwise Mutual Information (NPMI) coherence metric for topic modeling evaluation.

    This class calculates the NPMI coherence score, which measures the coherence of topics based on
    the normalized pointwise mutual information of word pairs.

    Key Features:
        - Measures topic coherence using normalized pointwise mutual information
        - Suitable for evaluating topic models
        - Handles input data as pandas DataFrames

    Usage:
        The NPMI metric can be used to evaluate topic modeling results:

        ```python
        from framework3.plugins.metrics.coherence import NPMI
        from framework3.base.base_types import XYData
        import pandas as pd
        import numpy as np

        # Assuming you have a DataFrame 'df' with your document-term matrix
        x_data = XYData(value=df)
        y_pred = np.array([['word1', 'word2', 'word3'], ['word4', 'word5', 'word6']])  # Example topics

        npmi_metric = NPMI()
        score = npmi_metric.evaluate(x_data, None, y_pred, f_vocab=df.columns)
        print(f"NPMI Score: {score}")
        ```

    Methods:
        evaluate(x_data: XYData, y_true: XYData | None, y_pred: XYData, **kwargs) -> Float | np.ndarray:
            Calculate the NPMI coherence score.

    Note:
        This metric requires the input data to be a pandas DataFrame. Ensure that your data
        is properly formatted before using this metric.
    """

    def evaluate(
        self,
        x_data: XYData,
        y_true: XYData | None,
        y_pred: XYData,
        **kwargs: Unpack[CoherenceEvaluateKwargs],
    ) -> Float | np.ndarray:
        """
        Calculate the NPMI coherence score.

        This method computes the NPMI coherence score for the given topics.

        Args:
            x_data (XYData): The input data, expected to be a pandas DataFrame.
            y_true (XYData | None): Not used for this metric, but required by the interface.
            y_pred (XYData): The predicted topics, typically a list of lists of words.
            **kwargs (Unpack[EvaluateKwargs]): Additional keyword arguments:
                - f_vocab (list): The vocabulary of the corpus.
                - topk (int): The number of top words to consider for each topic (default: 10).
                - processes (int): The number of processes to use for parallel computation (default: 1).

        Returns:
            Float | np.ndarray: The NPMI coherence score.

        Raises:
            Exception: If x_data is not a pandas DataFrame.

        Note:
            This method uses the Coherence class from framework3.plugins.metrics.utils.coherence internally.
        """
        f_vocab = kwargs.get("f_vocab")
        topk = kwargs.get("topk", 10)
        processes = kwargs.get("processes", 1)
        coherence = Coherence(
            f_vocab=f_vocab, topk=topk, processes=processes, measure="c_npmi"
        )
        if isinstance(x_data.value, pd.DataFrame):
            return coherence.evaluate(df=x_data.value, predicted=y_pred)
        else:
            raise Exception("x_data must be a pandas DataFrame")

evaluate(x_data, y_true, y_pred, **kwargs)

Calculate the NPMI coherence score.

This method computes the NPMI coherence score for the given topics.

Parameters:

Name Type Description Default
x_data XYData

The input data, expected to be a pandas DataFrame.

 required
y_true XYData | None

Not used for this metric, but required by the interface.

 required
y_pred XYData

The predicted topics, typically a list of lists of words.

 required
**kwargs Unpack[EvaluateKwargs]

Additional keyword arguments: - f_vocab (list): The vocabulary of the corpus. - topk (int): The number of top words to consider for each topic (default: 10). - processes (int): The number of processes to use for parallel computation (default: 1).

 {}

Returns:

Type Description
Float | ndarray

Float | np.ndarray: The NPMI coherence score.

Raises:

Type Description
Exception

If x_data is not a pandas DataFrame.
Note

This method uses the Coherence class from framework3.plugins.metrics.utils.coherence internally.

Source code in framework3/plugins/metrics/coherence.py 
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
def evaluate(
    self,
    x_data: XYData,
    y_true: XYData | None,
    y_pred: XYData,
    **kwargs: Unpack[CoherenceEvaluateKwargs],
) -> Float | np.ndarray:
    """
    Calculate the NPMI coherence score.

    This method computes the NPMI coherence score for the given topics.

    Args:
        x_data (XYData): The input data, expected to be a pandas DataFrame.
        y_true (XYData | None): Not used for this metric, but required by the interface.
        y_pred (XYData): The predicted topics, typically a list of lists of words.
        **kwargs (Unpack[EvaluateKwargs]): Additional keyword arguments:
            - f_vocab (list): The vocabulary of the corpus.
            - topk (int): The number of top words to consider for each topic (default: 10).
            - processes (int): The number of processes to use for parallel computation (default: 1).

    Returns:
        Float | np.ndarray: The NPMI coherence score.

    Raises:
        Exception: If x_data is not a pandas DataFrame.

    Note:
        This method uses the Coherence class from framework3.plugins.metrics.utils.coherence internally.
    """
    f_vocab = kwargs.get("f_vocab")
    topk = kwargs.get("topk", 10)
    processes = kwargs.get("processes", 1)
    coherence = Coherence(
        f_vocab=f_vocab, topk=topk, processes=processes, measure="c_npmi"
    )
    if isinstance(x_data.value, pd.DataFrame):
        return coherence.evaluate(df=x_data.value, predicted=y_pred)
    else:
        raise Exception("x_data must be a pandas DataFrame")

UCI

Bases: BaseMetric

UCI coherence metric for topic modeling evaluation.

This class calculates the UCI coherence score, which is based on pointwise mutual information (PMI) of all word pairs in a topic.

Key Features
  • Measures topic coherence using pointwise mutual information of word pairs
  • Suitable for evaluating topic models
  • Handles input data as pandas DataFrames
Usage

The UCI metric can be used to evaluate topic modeling results:

from framework3.plugins.metrics.coherence import UCI
from framework3.base.base_types import XYData
import pandas as pd
import numpy as np

# Assuming you have a DataFrame 'df' with your document-term matrix
x_data = XYData(value=df)
y_pred = np.array([['word1', 'word2', 'word3'], ['word4', 'word5', 'word6']])  # Example topics

uci_metric = UCI()
score = uci_metric.evaluate(x_data, None, y_pred, f_vocab=df.columns)
print(f"UCI Score: {score}")

Methods:

Name Description
evaluate

XYData, y_true: XYData | None, y_pred: XYData, **kwargs) -> Float | np.ndarray: Calculate the UCI coherence score.
Note

This metric requires the input data to be a pandas DataFrame. Ensure that your data is properly formatted before using this metric.

Source code in framework3/plugins/metrics/coherence.py 
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
@Container.bind()
class UCI(BaseMetric):
    """
    UCI coherence metric for topic modeling evaluation.

    This class calculates the UCI coherence score, which is based on pointwise mutual information (PMI)
    of all word pairs in a topic.

    Key Features:
        - Measures topic coherence using pointwise mutual information of word pairs
        - Suitable for evaluating topic models
        - Handles input data as pandas DataFrames

    Usage:
        The UCI metric can be used to evaluate topic modeling results:

        ```python
        from framework3.plugins.metrics.coherence import UCI
        from framework3.base.base_types import XYData
        import pandas as pd
        import numpy as np

        # Assuming you have a DataFrame 'df' with your document-term matrix
        x_data = XYData(value=df)
        y_pred = np.array([['word1', 'word2', 'word3'], ['word4', 'word5', 'word6']])  # Example topics

        uci_metric = UCI()
        score = uci_metric.evaluate(x_data, None, y_pred, f_vocab=df.columns)
        print(f"UCI Score: {score}")
        ```

    Methods:
        evaluate(x_data: XYData, y_true: XYData | None, y_pred: XYData, **kwargs) -> Float | np.ndarray:
            Calculate the UCI coherence score.

    Note:
        This metric requires the input data to be a pandas DataFrame. Ensure that your data
        is properly formatted before using this metric.
    """

    def evaluate(
        self,
        x_data: XYData,
        y_true: XYData | None,
        y_pred: XYData,
        **kwargs: Unpack[CoherenceEvaluateKwargs],
    ) -> Float | np.ndarray:
        """
        Calculate the UCI coherence score.

        This method computes the UCI coherence score for the given topics.

        Args:
            x_data (XYData): The input data, expected to be a pandas DataFrame.
            y_true (XYData | None): Not used for this metric, but required by the interface.
            y_pred (XYData): The predicted topics, typically a list of lists of words.
            **kwargs (Unpack[EvaluateKwargs]): Additional keyword arguments:
                - f_vocab (list): The vocabulary of the corpus.
                - topk (int): The number of top words to consider for each topic (default: 10).
                - processes (int): The number of processes to use for parallel computation (default: 1).

        Returns:
            Float | np.ndarray: The UCI coherence score.

        Raises:
            Exception: If x_data is not a pandas DataFrame.

        Note:
            This method uses the Coherence class from framework3.plugins.metrics.utils.coherence internally.
        """
        f_vocab = kwargs.get("f_vocab")
        topk = cast(int, kwargs.get("topk", 10))
        processes = cast(int, kwargs.get("processes", 1))
        coherence = Coherence(
            f_vocab=f_vocab, topk=topk, processes=processes, measure="c_uci"
        )
        if isinstance(x_data.value, pd.DataFrame):
            return coherence.evaluate(df=x_data.value, predicted=y_pred)
        else:
            raise Exception("x_data must be a pandas DataFrame")

evaluate(x_data, y_true, y_pred, **kwargs)

Calculate the UCI coherence score.

This method computes the UCI coherence score for the given topics.

Parameters:

Name Type Description Default
x_data XYData

The input data, expected to be a pandas DataFrame.

 required
y_true XYData | None

Not used for this metric, but required by the interface.

 required
y_pred XYData

The predicted topics, typically a list of lists of words.

 required
**kwargs Unpack[EvaluateKwargs]

Additional keyword arguments: - f_vocab (list): The vocabulary of the corpus. - topk (int): The number of top words to consider for each topic (default: 10). - processes (int): The number of processes to use for parallel computation (default: 1).

 {}

Returns:

Type Description
Float | ndarray

Float | np.ndarray: The UCI coherence score.

Raises:

Type Description
Exception

If x_data is not a pandas DataFrame.
Note

This method uses the Coherence class from framework3.plugins.metrics.utils.coherence internally.

Source code in framework3/plugins/metrics/coherence.py 
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
def evaluate(
    self,
    x_data: XYData,
    y_true: XYData | None,
    y_pred: XYData,
    **kwargs: Unpack[CoherenceEvaluateKwargs],
) -> Float | np.ndarray:
    """
    Calculate the UCI coherence score.

    This method computes the UCI coherence score for the given topics.

    Args:
        x_data (XYData): The input data, expected to be a pandas DataFrame.
        y_true (XYData | None): Not used for this metric, but required by the interface.
        y_pred (XYData): The predicted topics, typically a list of lists of words.
        **kwargs (Unpack[EvaluateKwargs]): Additional keyword arguments:
            - f_vocab (list): The vocabulary of the corpus.
            - topk (int): The number of top words to consider for each topic (default: 10).
            - processes (int): The number of processes to use for parallel computation (default: 1).

    Returns:
        Float | np.ndarray: The UCI coherence score.

    Raises:
        Exception: If x_data is not a pandas DataFrame.

    Note:
        This method uses the Coherence class from framework3.plugins.metrics.utils.coherence internally.
    """
    f_vocab = kwargs.get("f_vocab")
    topk = cast(int, kwargs.get("topk", 10))
    processes = cast(int, kwargs.get("processes", 1))
    coherence = Coherence(
        f_vocab=f_vocab, topk=topk, processes=processes, measure="c_uci"
    )
    if isinstance(x_data.value, pd.DataFrame):
        return coherence.evaluate(df=x_data.value, predicted=y_pred)
    else:
        raise Exception("x_data must be a pandas DataFrame")

UMASS

Bases: BaseMetric

UMass coherence metric for topic modeling evaluation.

This class calculates the UMass coherence score, which is based on document co-occurrence counts and a sliding window.

Key Features
  • Measures topic coherence using document co-occurrence
  • Suitable for evaluating topic models
  • Handles input data as pandas DataFrames
Usage

The UMASS metric can be used to evaluate topic modeling results:

from framework3.plugins.metrics.coherence import UMASS
from framework3.base.base_types import XYData
import pandas as pd
import numpy as np

# Assuming you have a DataFrame 'df' with your document-term matrix
x_data = XYData(value=df)
y_pred = np.array([['word1', 'word2', 'word3'], ['word4', 'word5', 'word6']])  # Example topics

umass_metric = UMASS()
score = umass_metric.evaluate(x_data, None, y_pred, f_vocab=df.columns)
print(f"UMass Score: {score}")

Methods:

Name Description
evaluate

XYData, y_true: XYData | None, y_pred: XYData, **kwargs) -> Float | np.ndarray: Calculate the UMass coherence score.
Note

This metric requires the input data to be a pandas DataFrame. Ensure that your data is properly formatted before using this metric.

Source code in framework3/plugins/metrics/coherence.py 
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
@Container.bind()
class UMASS(BaseMetric):
    """
    UMass coherence metric for topic modeling evaluation.

    This class calculates the UMass coherence score, which is based on document co-occurrence counts
    and a sliding window.

    Key Features:
        - Measures topic coherence using document co-occurrence
        - Suitable for evaluating topic models
        - Handles input data as pandas DataFrames

    Usage:
        The UMASS metric can be used to evaluate topic modeling results:

        ```python
        from framework3.plugins.metrics.coherence import UMASS
        from framework3.base.base_types import XYData
        import pandas as pd
        import numpy as np

        # Assuming you have a DataFrame 'df' with your document-term matrix
        x_data = XYData(value=df)
        y_pred = np.array([['word1', 'word2', 'word3'], ['word4', 'word5', 'word6']])  # Example topics

        umass_metric = UMASS()
        score = umass_metric.evaluate(x_data, None, y_pred, f_vocab=df.columns)
        print(f"UMass Score: {score}")
        ```

    Methods:
        evaluate(x_data: XYData, y_true: XYData | None, y_pred: XYData, **kwargs) -> Float | np.ndarray:
            Calculate the UMass coherence score.

    Note:
        This metric requires the input data to be a pandas DataFrame. Ensure that your data
        is properly formatted before using this metric.
    """

    def evaluate(
        self,
        x_data: XYData,
        y_true: XYData | None,
        y_pred: XYData,
        **kwargs: Unpack[CoherenceEvaluateKwargs],
    ) -> Float | np.ndarray:
        """
        Calculate the UMass coherence score.

        This method computes the UMass coherence score for the given topics.

        Args:
            x_data (XYData): The input data, expected to be a pandas DataFrame.
            y_true (XYData | None): Not used for this metric, but required by the interface.
            y_pred (XYData): The predicted topics, typically a list of lists of words.
            **kwargs (Unpack[EvaluateKwargs]): Additional keyword arguments:
                - f_vocab (list): The vocabulary of the corpus.
                - topk (int): The number of top words to consider for each topic (default: 10).
                - processes (int): The number of processes to use for parallel computation (default: 1).

        Returns:
            Float | np.ndarray: The UMass coherence score.

        Raises:
            Exception: If x_data is not a pandas DataFrame.

        Note:
            This method uses the Coherence class from framework3.plugins.metrics.utils.coherence internally.
        """
        f_vocab = kwargs.get("f_vocab")
        topk = kwargs.get("topk", 10)
        processes = kwargs.get("processes", 1)
        coherence = Coherence(
            f_vocab=f_vocab, topk=topk, processes=processes, measure="u_mass"
        )
        if isinstance(x_data.value, pd.DataFrame):
            return coherence.evaluate(df=x_data.value, predicted=y_pred)
        else:
            raise Exception("x_data must be a pandas DataFrame")

evaluate(x_data, y_true, y_pred, **kwargs)

Calculate the UMass coherence score.

This method computes the UMass coherence score for the given topics.

Parameters:

Name Type Description Default
x_data XYData

The input data, expected to be a pandas DataFrame.

 required
y_true XYData | None

Not used for this metric, but required by the interface.

 required
y_pred XYData

The predicted topics, typically a list of lists of words.

 required
**kwargs Unpack[EvaluateKwargs]

Additional keyword arguments: - f_vocab (list): The vocabulary of the corpus. - topk (int): The number of top words to consider for each topic (default: 10). - processes (int): The number of processes to use for parallel computation (default: 1).

 {}

Returns:

Type Description
Float | ndarray

Float | np.ndarray: The UMass coherence score.

Raises:

Type Description
Exception

If x_data is not a pandas DataFrame.
Note

This method uses the Coherence class from framework3.plugins.metrics.utils.coherence internally.

Source code in framework3/plugins/metrics/coherence.py 
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
def evaluate(
    self,
    x_data: XYData,
    y_true: XYData | None,
    y_pred: XYData,
    **kwargs: Unpack[CoherenceEvaluateKwargs],
) -> Float | np.ndarray:
    """
    Calculate the UMass coherence score.

    This method computes the UMass coherence score for the given topics.

    Args:
        x_data (XYData): The input data, expected to be a pandas DataFrame.
        y_true (XYData | None): Not used for this metric, but required by the interface.
        y_pred (XYData): The predicted topics, typically a list of lists of words.
        **kwargs (Unpack[EvaluateKwargs]): Additional keyword arguments:
            - f_vocab (list): The vocabulary of the corpus.
            - topk (int): The number of top words to consider for each topic (default: 10).
            - processes (int): The number of processes to use for parallel computation (default: 1).

    Returns:
        Float | np.ndarray: The UMass coherence score.

    Raises:
        Exception: If x_data is not a pandas DataFrame.

    Note:
        This method uses the Coherence class from framework3.plugins.metrics.utils.coherence internally.
    """
    f_vocab = kwargs.get("f_vocab")
    topk = kwargs.get("topk", 10)
    processes = kwargs.get("processes", 1)
    coherence = Coherence(
        f_vocab=f_vocab, topk=topk, processes=processes, measure="u_mass"
    )
    if isinstance(x_data.value, pd.DataFrame):
        return coherence.evaluate(df=x_data.value, predicted=y_pred)
    else:
        raise Exception("x_data must be a pandas DataFrame")

V

Bases: BaseMetric

V-measure coherence metric for topic modeling evaluation.

This class calculates the V-measure coherence score, which is based on a combination of homogeneity and completeness.

Key Features
  • Measures topic coherence using a combination of homogeneity and completeness
  • Suitable for evaluating topic models
  • Handles input data as pandas DataFrames
Usage

The V-measure metric can be used to evaluate topic modeling results:

from framework3.plugins.metrics.coherence import V
from framework3.base.base_types import XYData
import pandas as pd
import numpy as np

# Assuming you have a DataFrame 'df' with your document-term matrix
x_data = XYData(value=df)
y_pred = np.array([['word1', 'word2', 'word3'], ['word4', 'word5', 'word6']])  # Example topics

v_metric = V()
score = v_metric.evaluate(x_data, None, y_pred, f_vocab=df.columns)
print(f"V-measure Score: {score}")

Methods:

Name Description
evaluate

XYData, y_true: XYData | None, y_pred: XYData, **kwargs) -> Float | np.ndarray: Calculate the V-measure coherence score.
Note

This metric requires the input data to be a pandas DataFrame. Ensure that your data is properly formatted before using this metric.

Source code in framework3/plugins/metrics/coherence.py 
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
@Container.bind()
class V(BaseMetric):
    """
    V-measure coherence metric for topic modeling evaluation.

    This class calculates the V-measure coherence score, which is based on a combination of
    homogeneity and completeness.

    Key Features:
        - Measures topic coherence using a combination of homogeneity and completeness
        - Suitable for evaluating topic models
        - Handles input data as pandas DataFrames

    Usage:
        The V-measure metric can be used to evaluate topic modeling results:

        ```python
        from framework3.plugins.metrics.coherence import V
        from framework3.base.base_types import XYData
        import pandas as pd
        import numpy as np

        # Assuming you have a DataFrame 'df' with your document-term matrix
        x_data = XYData(value=df)
        y_pred = np.array([['word1', 'word2', 'word3'], ['word4', 'word5', 'word6']])  # Example topics

        v_metric = V()
        score = v_metric.evaluate(x_data, None, y_pred, f_vocab=df.columns)
        print(f"V-measure Score: {score}")
        ```

    Methods:
        evaluate(x_data: XYData, y_true: XYData | None, y_pred: XYData, **kwargs) -> Float | np.ndarray:
            Calculate the V-measure coherence score.

    Note:
        This metric requires the input data to be a pandas DataFrame. Ensure that your data
        is properly formatted before using this metric.
    """

    def evaluate(
        self,
        x_data: XYData,
        y_true: XYData | None,
        y_pred: XYData,
        **kwargs: Unpack[CoherenceEvaluateKwargs],
    ) -> Float | np.ndarray:
        """
        Calculate the V-measure coherence score.

        This method computes the V-measure coherence score for the given topics.

        Args:
            x_data (XYData): The input data, expected to be a pandas DataFrame.
            y_true (XYData | None): Not used for this metric, but required by the interface.
            y_pred (XYData): The predicted topics, typically a list of lists of words.
            **kwargs (Unpack[EvaluateKwargs]): Additional keyword arguments:
                - f_vocab (list): The vocabulary of the corpus.
                - topk (int): The number of top words to consider for each topic (default: 10).
                - processes (int): The number of processes to use for parallel computation (default: 1).

        Returns:
            Float | np.ndarray: The V-measure coherence score.

        Raises:
            Exception: If x_data is not a pandas DataFrame.

        Note:
            This method uses the Coherence class from framework3.plugins.metrics.utils.coherence internally.
        """
        f_vocab = kwargs.get("f_vocab")
        topk: int = cast(int, kwargs.get("topk", 10))
        processes: int = cast(int, kwargs.get("processes", 1))
        coherence = Coherence(
            f_vocab=f_vocab, topk=topk, processes=processes, measure="c_v"
        )
        if isinstance(x_data.value, pd.DataFrame):
            return coherence.evaluate(df=x_data.value, predicted=y_pred)
        else:
            raise Exception("x_data must be a pandas DataFrame")

evaluate(x_data, y_true, y_pred, **kwargs)

Calculate the V-measure coherence score.

This method computes the V-measure coherence score for the given topics.

Parameters:

Name Type Description Default
x_data XYData

The input data, expected to be a pandas DataFrame.

 required
y_true XYData | None

Not used for this metric, but required by the interface.

 required
y_pred XYData

The predicted topics, typically a list of lists of words.

 required
**kwargs Unpack[EvaluateKwargs]

Additional keyword arguments: - f_vocab (list): The vocabulary of the corpus. - topk (int): The number of top words to consider for each topic (default: 10). - processes (int): The number of processes to use for parallel computation (default: 1).

 {}

Returns:

Type Description
Float | ndarray

Float | np.ndarray: The V-measure coherence score.

Raises:

Type Description
Exception

If x_data is not a pandas DataFrame.
Note

This method uses the Coherence class from framework3.plugins.metrics.utils.coherence internally.

Source code in framework3/plugins/metrics/coherence.py 
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
def evaluate(
    self,
    x_data: XYData,
    y_true: XYData | None,
    y_pred: XYData,
    **kwargs: Unpack[CoherenceEvaluateKwargs],
) -> Float | np.ndarray:
    """
    Calculate the V-measure coherence score.

    This method computes the V-measure coherence score for the given topics.

    Args:
        x_data (XYData): The input data, expected to be a pandas DataFrame.
        y_true (XYData | None): Not used for this metric, but required by the interface.
        y_pred (XYData): The predicted topics, typically a list of lists of words.
        **kwargs (Unpack[EvaluateKwargs]): Additional keyword arguments:
            - f_vocab (list): The vocabulary of the corpus.
            - topk (int): The number of top words to consider for each topic (default: 10).
            - processes (int): The number of processes to use for parallel computation (default: 1).

    Returns:
        Float | np.ndarray: The V-measure coherence score.

    Raises:
        Exception: If x_data is not a pandas DataFrame.

    Note:
        This method uses the Coherence class from framework3.plugins.metrics.utils.coherence internally.
    """
    f_vocab = kwargs.get("f_vocab")
    topk: int = cast(int, kwargs.get("topk", 10))
    processes: int = cast(int, kwargs.get("processes", 1))
    coherence = Coherence(
        f_vocab=f_vocab, topk=topk, processes=processes, measure="c_v"
    )
    if isinstance(x_data.value, pd.DataFrame):
        return coherence.evaluate(df=x_data.value, predicted=y_pred)
    else:
        raise Exception("x_data must be a pandas DataFrame")

Overview

The Coherence Metrics module in framework3 provides evaluation metrics specifically designed for assessing the quality and interpretability of topic models. These metrics help in understanding how well the generated topics represent the underlying themes in a corpus of documents.

Available Coherence Metrics

NPMI (Normalized Pointwise Mutual Information)

The NPMI coherence metric is implemented in the NPMI class. It measures the semantic similarity between high-scoring words in the topic based on normalized pointwise mutual information.

Usage

from framework3.plugins.metrics.coherence import NPMI
from framework3.base.base_types import XYData

npmi_metric = NPMI(measure='c_npmi', topk=10, processes=1)
score = npmi_metric.evaluate(df, predicted)

Parameters: - measure: The coherence measure to use. Default is 'c_npmi'. - topk: The number of top words to consider for each topic. Default is 10. - processes: The number of processes to use for parallel computation. Default is 1.

How Coherence Metrics Work

Coherence metrics typically work by:

  1. Extracting the top N words from each topic.
  2. Calculating the semantic similarity between these words based on their co-occurrence in the corpus.
  3. Aggregating these similarities to produce a single coherence score for each topic or for the entire model.

The NPMI metric specifically:

  1. Calculates the pointwise mutual information (PMI) between pairs of words.
  2. Normalizes the PMI scores to account for the frequency of individual words.
  3. Averages these normalized scores to produce the final coherence score.

Comprehensive Example: Evaluating a Topic Model

Here's an example of how to use the NPMI coherence metric to evaluate a topic model:

import pandas as pd
from framework3.plugins.filters.topic_modeling.lda import LDAPlugin
from framework3.plugins.metrics.coherence import NPMI
from framework3.base.base_types import XYData

# Assume we have a DataFrame 'df' with a 'text' column containing our documents
df = pd.DataFrame({'text': ["This is document 1", "This is document 2", "Another document here"]})

# Create XYData object
X_data = XYData(_hash='X_data', _path='/tmp', _value=df['text'].values)

# Create and fit the LDA model
lda_model = LDAPlugin(n_components=5, random_state=42)
lda_model.fit(X_data)

# Get topic-word distributions
topic_word_dist = lda_model.get_topic_word_dist()

# Initialize NPMI metric
npmi_metric = NPMI(measure='c_npmi', topk=10, processes=1)

# Evaluate coherence
coherence_score = npmi_metric.evaluate(df, (None, topic_word_dist, None))

print(f"NPMI Coherence Score: {coherence_score}")

This example demonstrates how to:

  1. Prepare your text data
  2. Create XYData objects for use with framework3
  3. Train an LDA topic model
  4. Extract topic-word distributions
  5. Initialize and compute the NPMI coherence metric
  6. Print the evaluation result

Best Practices

  1. Multiple Runs: Topic modeling algorithms often have a random component. Run your model multiple times and average the coherence scores for more stable results.

  2. Number of Topics: Use coherence metrics to help determine the optimal number of topics for your model. Try different numbers of topics and compare their coherence scores.

  3. Preprocessing: The quality of your preprocessing can significantly affect coherence scores. Ensure your text is properly cleaned and tokenized.

  4. Interpretation: Remember that while higher coherence scores generally indicate better topic quality, they should be interpreted in conjunction with qualitative analysis of the topics.

  5. Comparison: Use coherence metrics to compare different topic modeling approaches (e.g., LDA vs. NMF) on the same dataset.

  6. Domain Knowledge: Always interpret coherence scores in the context of your domain knowledge and the specific goals of your topic modeling task.

  7. Visualization: Complement coherence metrics with visualization techniques (like pyLDAvis) to get a better understanding of your topic model results.

  8. Parameter Tuning: Use coherence scores to guide the tuning of your topic model's parameters (e.g., alpha and beta in LDA).

Conclusion

The Coherence Metrics module in framework3 provides essential tools for evaluating the quality of topic models. By using these metrics in combination with other framework3 components, you can gain valuable insights into your model's performance and interpretability. The example demonstrates how easy it is to compute and interpret these metrics within the framework3 ecosystem, enabling you to make informed decisions about your topic modeling approach.