SparkR is just a wrapper on Scala implementations. Are you just looking for setting family = multinomial on LogisticRegression ? Sure it's there in the scala API
On Mon, Feb 14, 2022, 11:50 AM Surya Rajaraman Iyer <si...@medallia.com> wrote: > Hi Team, > > I am using a multinomial regression in Spark Scala. I want to generate the > coefficient and p-values for every category. > > For example, given two variables salary group (dependent variable) and age > group (Independent variable) > > salary-group: 10,000-, 10,000-100,000, 100,000+ > age-group: 30-, 30-40, 40+ > > I am looking to get an output like > > With 10,000- as baseline, get the coefficients and pvalues for each > category. in the salary group > > 10,000-100,000, > coefficient Pvalue > Intercept .. .. > > age group > 30-40 .. .. > 40+ .. .. > 30- 0 0 > > 100,000+ coefficient Pvalue > Intercept .. .. > > age group > 30-40 .. .. > 40+ .. .. > 30- 0 0 > To do this, I am forced to use glm with binomial family twice. In order to > parallelize it, I am using thread pools which doesn't seem ideal. > > Do you think there is a way to do multinomial logit in spark scala.I do > see it in spark R : https://rdrr.io/cran/SparkR/man/spark.logit.html > > Is there a spark way to make the glms parallel? Something like:- > > SparkLogisticRegressionResult glm (df: DataFrame) { > } > > dfs : Seq[df] > dfs.map(glm) > > > Thanks a lot for the help! > > Regards, > Surya, > > Confidentiality Notice: This email and any files transmitted with it are > confidential and intended solely for the use of the individual or entity to > whom they are addressed. Additionally, this email and any files > transmitted with it may not be disseminated, distributed or copied. Please > notify the sender immediately by email if you have received this email by > mistake and delete this email from your system. If you are not the intended > recipient, you are notified that disclosing, copying, distributing or > taking any action in reliance on the contents of this information is > strictly prohibited. > > [image: > http://www.medallia.com/gartner-report/?source=Marketing%20-%20Email&utm_campaign=FY22Q4_NA_Gartner_MQ_VoC_Campaign&utm_medium=email&utm_source=email-signature&utm_content=report&utm_term=medallia-named-a-leader] > <http://www.medallia.com/gartner-report/?source=Marketing%20-%20Email&utm_campaign=FY22Q4_NA_Gartner_MQ_VoC_Campaign&utm_medium=email&utm_source=email-signature&utm_content=report&utm_term=medallia-named-a-leader> >
