My two cents (As a user/consumer)… I have been following & using Spark in financial services before version 1 and before it migrated questions from Google Groups to apache mailing lists (which was a shame ☹ ).
SO: There has been some momentum lately on SO, but as questions were not “monitored/answered” by Spark experts, the motivation of posting a question was low and in turn the quality of questions as well. As most of us know, SO is usually the first place to look for info and can greatly reduce the need to turn to user/dev groups so it would be great if there was more attention to it. Spark mailing lists: As the consensus appears to be, questions tend to get lost if not picked-up within 1-2 days. Re-sending the same question feels “abusive” to me so would then give up. Provided that a good question takes time, putting effort in a question that can easily be ignored results to mailing a “bad” question (see what happens?) or no question at all. As you have probably observed, a few users will mail a question to “dev” with “…no answers in user list…” as they incorrectly assume that no-one can answer their question. JIRA: I find that “issues” are being quite aggressively closed down. I’ve seen this twice (one I reported myself and found the second ticket while looking for a solution) and for this reason it doesn’t encourage users spending the time and effort to use. Personally, I also feel that there is some bias on what is in-scope and out-of-scope. My preference would be that SO would be the first place that someone would post a question. If a few “experts” are found regularly answering questions, eventually Spark users will start using it more and reduce “user” load by easily finding previous answers (or SO community marking a duplicates). The same “experts” can also encourage users to “escalate” to JIRA, dev/user groups once a question has been properly filtered which is quite common. PS. Personally, I would not follow any “bespoke/external” process on SO E.g. down-voting on SO for any other reason that being a bad question as per SO rules. From: Matei Zaharia [mailto:matei.zaha...@gmail.com] Sent: 07 November 2016 07:45 To: assaf.mendelson Cc: dev@spark.apache.org Subject: Re: Handling questions in the mailing lists Even for the mailing list, I'd love to have a short set of instructions on how to submit your questions (maybe on http://spark.apache.org/community.html<https://urldefense.proofpoint.com/v2/url?u=http-3A__spark.apache.org_community.html&d=DQMFaQ&c=dCBwIlVXJsYZrY6gpNt0LA&r=B8E4n9FrSS85mPCi6Mfs7cyEPQnVrpcQ1zeB-JKws6A&m=Vf-yZoTpLgwZzwUCoQTMr4UFD_R0nx0naxh_SWUHfho&s=MIQDl3ZflIuyNs62JLog9_vi0dD4xyo96x2w7XwGV3w&e=> or maybe in the welcome email when you subscribe). It would be great if someone added that. After all, we have such instructions for contributing PRs, for example. Matei On Nov 6, 2016, at 11:09 PM, assaf.mendelson <assaf.mendel...@rsa.com<mailto:assaf.mendel...@rsa.com>> wrote: There are other options as well. For example hosting an answerhub (www.answerhub.com<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.answerhub.com_&d=DQMFaQ&c=dCBwIlVXJsYZrY6gpNt0LA&r=B8E4n9FrSS85mPCi6Mfs7cyEPQnVrpcQ1zeB-JKws6A&m=Vf-yZoTpLgwZzwUCoQTMr4UFD_R0nx0naxh_SWUHfho&s=2oSovyR4k9m576OtymFnf4nQ4Xksk94HX543bDeEVQI&e=>) or other similar separate Q&A service. BTW, I believe the main issue is not how opinionated people are but who is answering questions. Today there are already people asking (and getting answers) on SO (including myself). The problem is that many people do not go to SO. The problem I see is how to “bump” up questions which are not being answered to someone more likely to be able to answer them. Simple questions can be answered by many people, many of them even newbies who ran into the issue themselves. The main issue is that the more complex the question, the less people there are who can answer it and those people’s bandwidth is already clogged by other questions. We could for example try to create tags on SO for “basic questions”, “medium”, “advanced”. Provide guidelines to ask first on basic, if not answered after X days then add the medium tag etc. Downvote people who don’t go by the process. This would mean that committers for example can look at advanced only tag and have a manageable number of questions they can help with while others can answer medium and basic. I agree that some things are not good for SO. Basically stuff which asks for opinion is such but most cases in the mailing list are either “how do I solve this bug” or “how do I do X”. Either of those two are good for SO. Assaf. From: rxin [via Apache Spark Developers List] [mailto:ml-node+[hidden email]<x-msg://40/user/SendEmail.jtp?type=node&node=19758&i=0>] Sent: Monday, November 07, 2016 8:33 AM To: Mendelson, Assaf Subject: Re: Handling questions in the mailing lists This is an excellent point. If we do go ahead and feature SO as a way for users to ask questions more prominently, as someone who knows SO very well, would you be willing to help write a short guideline (ideally the shorter the better, which makes it hard) to direct what goes to user@ and what goes to SO? On Sun, Nov 6, 2016 at 9:54 PM, Maciej Szymkiewicz <[hidden email]<x-msg://40/user/SendEmail.jtp?type=node&node=19757&i=0>> wrote: Damn, I always thought that mailing list is only for nice and welcoming people and there is nothing to do for me here >:) To be serious though, there are many questions on the users list which would fit just fine on SO but it is not true in general. There are dozens of questions which are to broad, opinion based, ask for external resources and so on. If you want to direct users to SO you have to help them to decide if it is the right channel. Otherwise it will just create a really bad experience for both seeking help and active answerers. Former ones will be downvoted and bashed, latter ones will have to deal with handling all the junk and the number of active Spark users with moderation privileges is really low (with only Massg and me being able to directly close duplicates). Believe me, I've seen this before. On 11/07/2016 05:08 AM, Reynold Xin wrote: You have substantially underestimated how opinionated people can be on mailing lists too :) On Sunday, November 6, 2016, Maciej Szymkiewicz <[hidden email]<x-msg://40/user/SendEmail.jtp?type=node&node=19757&i=1>> wrote: You have to remember that Stack Overflow crowd (like me) is highly opinionated, so many questions, which could be just fine on the mailing list, will be quickly downvoted and / or closed as off-topic. Just saying... -- Best, Maciej On 11/07/2016 04:03 AM, Reynold Xin wrote: OK I've checked on the ASF member list (which is private so there is no public archive). It is not against any ASF rule to recommend StackOverflow as a place for users to ask questions. I don't think we can or should delete the existing user@spark list either, but we can certainly make SO more visible than it is. On Wed, Nov 2, 2016 at 10:21 AM, Reynold Xin <[hidden email]<x-msg://40/user/SendEmail.jtp?type=node&node=19757&i=2>> wrote: Actually after talking with more ASF members, I believe the only policy is that development decisions have to be made and announced on ASF properties (dev list or jira), but user questions don't have to. I'm going to double check this. If it is true, I would actually recommend us moving entirely over the Q&A part of the user list to stackoverflow, or at least make that the recommended way rather than the existing user list which is not very scalable. On Wednesday, November 2, 2016, Nicholas Chammas <[hidden email]<x-msg://40/user/SendEmail.jtp?type=node&node=19757&i=3>> wrote: We’ve discussed several times upgrading our communication tools, as far back as 2014 and maybe even before that too. The bottom line is that we can’t due to ASF rules requiring the use of ASF-managed mailing lists. For some history, see this discussion: 1. https://mail-archives.apache.org/mod_mbox/spark-user/201412.mbox/%3CCAOhmDzfL2COdysV8r5hZN8f=NqXM=f=oY5NO2dHWJ_kVEoP+Ng@...%3E<https://urldefense.proofpoint.com/v2/url?u=https-3A__mail-2Darchives.apache.org_mod-5Fmbox_spark-2Duser_201412.mbox_-253CCAOhmDzfL2COdysV8r5hZN8f-3DNqXM-3Df-3DoY5NO2dHWJ-5FkVEoP-2BNg-40mail.gmail.com-253E&d=DQMFaQ&c=dCBwIlVXJsYZrY6gpNt0LA&r=B8E4n9FrSS85mPCi6Mfs7cyEPQnVrpcQ1zeB-JKws6A&m=Vf-yZoTpLgwZzwUCoQTMr4UFD_R0nx0naxh_SWUHfho&s=fILmWaylBzYeV5-XRmdm75cBbKG57kiU81cArNLLbdA&e=> 2. https://mail-archives.apache.org/mod_mbox/spark-user/201501.mbox/%3CCAOhmDzec1JdsXQq3dDwAv7eLnzRidSkrsKKG0xKw=TKTxY_sYw@...%3E<https://urldefense.proofpoint.com/v2/url?u=https-3A__mail-2Darchives.apache.org_mod-5Fmbox_spark-2Duser_201501.mbox_-253CCAOhmDzec1JdsXQq3dDwAv7eLnzRidSkrsKKG0xKw-3DTKTxY-5FsYw-40mail.gmail.com-253E&d=DQMFaQ&c=dCBwIlVXJsYZrY6gpNt0LA&r=B8E4n9FrSS85mPCi6Mfs7cyEPQnVrpcQ1zeB-JKws6A&m=Vf-yZoTpLgwZzwUCoQTMr4UFD_R0nx0naxh_SWUHfho&s=_snNLu3ds5DSqCrMJ30_tq_qhaCPD6I72Sc25p0idmY&e=> (It’s ironic that it’s difficult to follow the past discussion on why we can’t change our official communication tools due to those very tools…) Nick On Wed, Nov 2, 2016 at 12:24 PM Ricardo Almeida <[hidden email]<x-msg://40/user/SendEmail.jtp?type=node&node=19757&i=4>> wrote: I fell Assaf point is quite relevant if we want to move this project forward from the Spark user perspective (as I do). In fact, we're still using 20th century tools (mailing lists) with some add-ons (like Stack Overflow). As usually, Sean and Cody's contributions are very to the point. I fell it is indeed a matter of of culture (hard to enforce) and tools (much easier). Isn't it? On 2 November 2016 at 16:36, Cody Koeninger <[hidden email]<x-msg://40/user/SendEmail.jtp?type=node&node=19757&i=5>> wrote: So concrete things people could do - users could tag subject lines appropriately to the component they're asking about - contributors could monitor user@ for tags relating to components they've worked on. I'd be surprised if my miss rate for any mailing list questions well-labeled as Kafka was higher than 5% - committers could be more aggressive about soliciting and merging PRs to improve documentation. It's a lot easier to answer even poorly-asked questions with a link to relevant docs. On Wed, Nov 2, 2016 at 7:39 AM, Sean Owen <[hidden email]<x-msg://40/user/SendEmail.jtp?type=node&node=19757&i=6>> wrote: > There's already reviews@ and issues@. dev@ is for project development itself > and I think is OK. You're suggesting splitting up user@ and I sympathize > with the motivation. Experience tells me that we'll have a beginner@ that's > then totally ignored, and people will quickly learn to post to advanced@ to > get attention, and we'll be back where we started. Putting it in JIRA > doesn't help. I don't think this a problem that is merely down to lack of > process. It actually requires cultivating a culture change on the community > list. > > On Wed, Nov 2, 2016 at 12:11 PM Mendelson, Assaf <[hidden > email]<x-msg://40/user/SendEmail.jtp?type=node&node=19757&i=7>> > wrote: >> >> What I am suggesting is basically to fix that. >> >> For example, we might say that mailing list A is only for voting, mailing >> list B is only for PR and have something like stack overflow for developer >> questions (I would even go as far as to have beginner, intermediate and >> advanced mailing list for users and beginner/advanced for dev). >> >> >> >> This can easily be done using stack overflow tags, however, that would >> probably be harder to manage. >> >> Maybe using special jira tags and manage it in jira? >> >> >> >> Anyway as I said, the main issue is not user questions (except maybe >> advanced ones) but more for dev questions. It is so easy to get lost in the >> chatter that it makes it very hard for people to learn spark internals… >> >> Assaf. >> >> >> >> From: Sean Owen [mailto:[hidden >> email]<x-msg://40/user/SendEmail.jtp?type=node&node=19757&i=8>] >> Sent: Wednesday, November 02, 2016 2:07 PM >> To: Mendelson, Assaf; [hidden >> email]<x-msg://40/user/SendEmail.jtp?type=node&node=19757&i=9> >> Subject: Re: Handling questions in the mailing lists >> >> >> >> I think that unfortunately mailing lists don't scale well. This one has >> thousands of subscribers with different interests and levels of experience. >> For any given person, most messages will be irrelevant. I also find that a >> lot of questions on user@ are not well-asked, aren't an SSCCE >> (http://sscce.org/<https://urldefense.proofpoint.com/v2/url?u=http-3A__sscce.org_&d=DQMFaQ&c=dCBwIlVXJsYZrY6gpNt0LA&r=B8E4n9FrSS85mPCi6Mfs7cyEPQnVrpcQ1zeB-JKws6A&m=Vf-yZoTpLgwZzwUCoQTMr4UFD_R0nx0naxh_SWUHfho&s=BhbHzFmq52GPq1-vnJpo1WYJSivxTYV2DTqLE2lcomU&e=>), >> not something most people are going to bother replying >> to even if they could answer. I almost entirely ignore user@ because there >> are higher-priority channels like PRs to deal with, that already have >> hundreds of messages per day. This is why little of it gets an answer -- too >> noisy. >> >> >> >> We have to have official mailing lists, in any event, to have some >> official channel for things like votes and announcements. It's not wrong to >> ask questions on user@ of course, but a lot of the questions I see could >> have been answered with research of existing docs or looking at the code. I >> think that given the scale of the list, it's not wrong to assert that this >> is sort of a prerequisite for asking thousands of people to answer one's >> question. But we can't enforce that. >> >> >> >> The situation will get better to the extent people ask better questions, >> help other people ask better questions, and answer good questions. I'd >> encourage anyone feeling this way to try to help along those dimensions. >> >> >> >> >> >> >> >> >> >> >> >> On Wed, Nov 2, 2016 at 11:32 AM assaf.mendelson <[hidden >> email]<x-msg://40/user/SendEmail.jtp?type=node&node=19757&i=10>> >> wrote: >> >> Hi, >> >> I know this is a little off topic but I wanted to raise an issue about >> handling questions in the mailing list (this is true both for the user >> mailing list and the dev but since there are other options such as stack >> overflow for user questions, this is more problematic in dev). >> >> Let’s say I ask a question (as I recently did). Unfortunately this was >> during spark summit in Europe so probably people were busy. In any case no >> one answered. >> >> The problem is, that if no one answers very soon, the question will almost >> certainly remain unanswered because new messages will simply drown it. >> >> >> >> This is a common issue not just for questions but for any comment or idea >> which is not immediately picked up. >> >> >> >> I believe we should have a method of handling this. >> >> Generally, I would say these types of things belong in stack overflow, >> after all, the way it is built is perfect for this. More seasoned spark >> contributors and committers can periodically check out unanswered questions >> and answer them. >> >> The problem is that stack overflow (as well as other targets such as the >> databricks forums) tend to have a more user based orientation. This means >> that any spark internal question will almost certainly remain unanswered. >> >> >> >> I was wondering if we could come up with a solution for this. >> >> >> >> Assaf. >> >> >> >> >> >> ________________________________ >> >> View this message in context: Handling questions in the mailing lists >> Sent from the Apache Spark Developers List mailing list archive at >> Nabble.com<https://urldefense.proofpoint.com/v2/url?u=http-3A__nabble.com&d=DQMFaQ&c=dCBwIlVXJsYZrY6gpNt0LA&r=B8E4n9FrSS85mPCi6Mfs7cyEPQnVrpcQ1zeB-JKws6A&m=Vf-yZoTpLgwZzwUCoQTMr4UFD_R0nx0naxh_SWUHfho&s=DOSXyWQ25VrvEJ61e9vezaFFqQ6ERTNkf2btm8y3JEA&e=>. --------------------------------------------------------------------- To unsubscribe e-mail: [hidden email]<x-msg://40/user/SendEmail.jtp?type=node&node=19757&i=11>rg ________________________________ If you reply to this email, your message will be added to the discussion below: http://apache-spark-developers-list.1001551.n3.nabble.com/Handling-questions-in-the-mailing-lists-tp19690p19757.html<https://urldefense.proofpoint.com/v2/url?u=http-3A__apache-2Dspark-2Ddevelopers-2Dlist.1001551.n3.nabble.com_Handling-2Dquestions-2Din-2Dthe-2Dmailing-2Dlists-2Dtp19690p19757.html&d=DQMFaQ&c=dCBwIlVXJsYZrY6gpNt0LA&r=B8E4n9FrSS85mPCi6Mfs7cyEPQnVrpcQ1zeB-JKws6A&m=Vf-yZoTpLgwZzwUCoQTMr4UFD_R0nx0naxh_SWUHfho&s=ZX7C3jXG0WAWbui6GXRkT15WDj5s6Yb9U_uCYr0p7Ew&e=> To start a new topic under Apache Spark Developers List, email [hidden email]<x-msg://40/user/SendEmail.jtp?type=node&node=19758&i=1> To unsubscribe from Apache Spark Developers List, click here. NAML<https://urldefense.proofpoint.com/v2/url?u=http-3A__apache-2Dspark-2Ddevelopers-2Dlist.1001551.n3.nabble.com_template_NamlServlet.jtp-3Fmacro-3Dmacro-5Fviewer-26id-3Dinstant-5Fhtml-2521nabble-253Aemail.naml-26base-3Dnabble.naml.namespaces.BasicNamespace-2Dnabble.view.web.template.NabbleNamespace-2Dnabble.view.web.template.NodeNamespace-26breadcrumbs-3Dnotify-5Fsubscribers-2521nabble-253Aemail.naml-2Dinstant-5Femails-2521nabble-253Aemail.naml-2Dsend-5Finstant-5Femail-2521nabble-253Aemail.naml&d=DQMFaQ&c=dCBwIlVXJsYZrY6gpNt0LA&r=B8E4n9FrSS85mPCi6Mfs7cyEPQnVrpcQ1zeB-JKws6A&m=Vf-yZoTpLgwZzwUCoQTMr4UFD_R0nx0naxh_SWUHfho&s=ZSjjNhkCGzaVsA9UhbpYXaKujat6vR7r6SdBFmyzWdc&e=> ________________________________ View this message in context: RE: Handling questions in the mailing lists<https://urldefense.proofpoint.com/v2/url?u=http-3A__apache-2Dspark-2Ddevelopers-2Dlist.1001551.n3.nabble.com_Handling-2Dquestions-2Din-2Dthe-2Dmailing-2Dlists-2Dtp19690p19758.html&d=DQMFaQ&c=dCBwIlVXJsYZrY6gpNt0LA&r=B8E4n9FrSS85mPCi6Mfs7cyEPQnVrpcQ1zeB-JKws6A&m=Vf-yZoTpLgwZzwUCoQTMr4UFD_R0nx0naxh_SWUHfho&s=ElcVsDry_U8-DCD_Awaa6kExNKJo0gQ6Dpbp0HNMUUI&e=> Sent from the Apache Spark Developers List mailing list archive<https://urldefense.proofpoint.com/v2/url?u=http-3A__apache-2Dspark-2Ddevelopers-2Dlist.1001551.n3.nabble.com_&d=DQMFaQ&c=dCBwIlVXJsYZrY6gpNt0LA&r=B8E4n9FrSS85mPCi6Mfs7cyEPQnVrpcQ1zeB-JKws6A&m=Vf-yZoTpLgwZzwUCoQTMr4UFD_R0nx0naxh_SWUHfho&s=dXRHr7SeGMH3ksDwIya1xGEJGhVRQU4TYLf3dun_L5k&e=> at Nabble.com<https://urldefense.proofpoint.com/v2/url?u=http-3A__nabble.com&d=DQMFaQ&c=dCBwIlVXJsYZrY6gpNt0LA&r=B8E4n9FrSS85mPCi6Mfs7cyEPQnVrpcQ1zeB-JKws6A&m=Vf-yZoTpLgwZzwUCoQTMr4UFD_R0nx0naxh_SWUHfho&s=DOSXyWQ25VrvEJ61e9vezaFFqQ6ERTNkf2btm8y3JEA&e=>. This e-mail (including any attachments) is private and confidential, may contain proprietary or privileged information and is intended for the named recipient(s) only. Unintended recipients are strictly prohibited from taking action on the basis of information in this e-mail and must contact the sender immediately, delete this e-mail (and all attachments) and destroy any hard copies. Nomura will not accept responsibility or liability for the accuracy or completeness of, or the presence of any virus or disabling code in, this e-mail. If verification is sought please request a hard copy. Any reference to the terms of executed transactions should be treated as preliminary only and subject to formal written confirmation by Nomura. Nomura reserves the right to retain, monitor and intercept e-mail communications through its networks (subject to and in accordance with applicable laws). No confidentiality or privilege is waived or lost by Nomura by any mistransmission of this e-mail. Any reference to "Nomura" is a reference to any entity in the Nomura Holdings, Inc. group. Please read our Electronic Communications Legal Notice which forms part of this e-mail: http://www.Nomura.com/email_disclaimer.htm
