Object

com.enriquegrodrigo.spark.crowd.methods

MajorityVoting

Related Doc: package methods

Permalink

object MajorityVoting

Provides functions for transforming an annotation dataset into a standard label dataset using the majority voting approach

This object provides several functions for using majority voting style algorithms over annotations datasets (spark datasets with types types.BinaryAnnotation, types.MulticlassAnnotation, or types.RealAnnotation). For discrete types (types.BinaryAnnotation, types.MulticlassAnnotation) the method uses the most frequent class. For continuous types, the mean is used.

The object also provides methods for estimating the probability of a class for the discrete type, computing, for the binary case, the proportion of the positive class and, for the multiclass case, the proportion of each of the classes.

The next example can be found in the examples folder of the project.

Example: 

  1. import com.enriquegrodrigo.spark.crowd.methods.MajorityVoting
import com.enriquegrodrigo.spark.crowd.types._
val exampleFile = "data/binary-ann.parquet"
val exampleFileMulti = "data/multi-ann.parquet"
val exampleFileCont = "data/cont-ann.parquet"
val exampleDataBinary = spark.read.parquet(exampleFile).as[BinaryAnnotation]
val exampleDataMulti = spark.read.parquet(exampleFileMulti).as[MulticlassAnnotation]
val exampleDataCont = spark.read.parquet(exampleFileCont).as[RealAnnotation]
//Applying the learning algorithm
//Binary class
val muBinary = MajorityVoting.transformBinary(exampleDataBinary)
val muBinaryProb = MajorityVoting.transformSoftBinary(exampleDataBinary)
//Multiclass
val muMulticlass = MajorityVoting.transformMulticlass(exampleDataMulti)
val muMulticlassProb = MajorityVoting.transformSoftMulti(exampleDataMulti)
//Continuous case
val muCont = MajorityVoting.transformReal(exampleDataCont)
Version

0.1.3

Linear Supertypes
AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. MajorityVoting
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any

  2. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any

  3. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any

  4. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any

  5. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  6. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef

  7. def equals(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any

  8. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )

  9. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any

  10. def hashCode(): Int

    Permalink
    Definition Classes
    AnyRef → Any

  11. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any

  12. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef

  13. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef

  14. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef

  15. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef

  16. def toString(): String

    Permalink
    Definition Classes
    AnyRef → Any

  17. def transformBinary(dataset: Dataset[BinaryAnnotation]): Dataset[BinaryLabel]

    Permalink

    Obtains the most frequent class (0 or 1) for types.BinaryAnnotation datasets

    Obtains the most frequent class (0 or 1) for types.BinaryAnnotation datasets

    dataset

    The annotations dataset (spark Dataset of type types.BinaryAnnotation) to be aggregated

  18. def transformMulticlass(dataset: Dataset[MulticlassAnnotation]): Dataset[MulticlassLabel]

    Permalink

    Obtain the most frequent class for each example of the a types.MulticlassAnnotation dataset.

    Obtain the most frequent class for each example of the a types.MulticlassAnnotation dataset.

    dataset

    The annotations dataset (spark Dataset of type types.MulticlassAnnotation) to be aggregated

  19. def transformReal(dataset: Dataset[RealAnnotation]): Dataset[RealLabel]

    Permalink

    Obtain the mean of the annotations for each example from a types.RealAnnotation.

    Obtain the mean of the annotations for each example from a types.RealAnnotation.

    dataset

    The annotations dataset (spark Dataset of type types.RealAnnotation) to be aggregated

  20. def transformSoftBinary(dataset: Dataset[BinaryAnnotation]): Dataset[BinarySoftLabel]

    Permalink

    Obtains probability of the class being positive for types.BinaryAnnotation datasets

    Obtains probability of the class being positive for types.BinaryAnnotation datasets

    dataset

    The annotations dataset (spark Dataset of type types.BinaryAnnotation) to be aggregated

  21. def transformSoftMulti(dataset: Dataset[MulticlassAnnotation]): Dataset[MulticlassSoftProb]

    Permalink

    Obtain a list of datasets resulting of applying transformSoftBinary to each class against the others (One vs All) on a types.MulticlassAnnotation dataset.

    Obtain a list of datasets resulting of applying transformSoftBinary to each class against the others (One vs All) on a types.MulticlassAnnotation dataset.

    It supposes classes go from 0 to nClasses. For example, for a three class problem, there should be classes {0,1,2}.

    dataset

    The annotations dataset (spark Dataset of type types.MulticlassAnnotation) to be aggregated

  22. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  23. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  24. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Inherited from AnyRef

Inherited from Any

Ungrouped