Re: Cross Join
Spark 2.0 is EOL. Can you try 2.3 or 2.4? On Thu, Mar 21, 2019 at 10:23 AM asma zgolli wrote: > Hello , > > I need to cross my data and i'm executing a cross join on two dataframes . > > C = A.crossJoin(B) > A has 50 records > B has 5 records > > the result im getting with spark 2.0 is a dataframe C having 50 records. > > only the first row from B was added to C. > > Is that a bug in Spark? > > Asma ZGOLLI > > PhD student in data engineering - computer science > >
Re: [build system] jenkins wedged again, rebooting master node
quick update: since kicking httpd on the jenkins master "fixes" the GUI hanging, i set up a cron job to restart httpd 4 times per day. this is not the final solution, but will definitely help over the weekend as i'm heading out of town. shane On Fri, Mar 22, 2019 at 9:50 AM shane knapp wrote: > i was right to not hold my breath... while my apache changes seem to have > helped a bit, things are still slowing down after 10-12 hours. > > i have a few other things i can look at, and will get as much done as > possible before the weekend. serious troubleshooting will begin anew > monday. > > apologies again, > > shane > > On Thu, Mar 21, 2019 at 12:54 PM shane knapp wrote: > >> i tweaked some apache settings (MaxClients increased to fix an error i >> found buried in the logs, and added 'retry' and 'acquire' to the reverse >> proxy settings to hopefully combat the dreaded 502 response), restarted >> httpd and things actually seem quite snappy right now! >> >> i'm not holding my breath, however... only time will tell. >> >> On Tue, Mar 19, 2019 at 7:18 AM Imran Rashid wrote: >> >>> seems wedged again? >>> >>> sorry for the bad news Shane, thanks for all the work on fixing it >>> >>> On Mon, Mar 18, 2019 at 4:02 PM shane knapp wrote: >>> ok, i dug through the logs and noticed that rsyslogd was dropping messages to do imuxsock being spammed by postfix... which i then tracked down to our installation of fail2ban being incorrectly configured and attempting to send IP ban/unban status emails to 'em...@example.com'. since we're a university, and especially one w/a reputation like ours, we are constantly under attack. the logs of the attempted dictionary attacks would astound you in their size and scope. since we have so many ban/unban actions happening for all of these unique IP address, each of which generates an email that was directed to an invalid address, we ended up w/well over 100M of plain-text messages waiting in the mail queue. postfix was continually trying to send these messages, which was causing the system to behave strangely, including breaking rsyslogd. so, i disabled email reports in fail2ban, restarted the impacted services, picked my sysadmin's brain and then purged the mail queue (when was the last time anyone actually used postfix?). jenkins now seems to be behaving (maybe?). i'm not entirely sure that this will fix the strange GUI hangs, but all reports i found on stackoverflow and other sites detail strange system behavior across the board when rsyslogd starts dropping messages. at the very least we won't be (potentially) losing system-level log messages anymore, which might actually help me track down what's happening if jenkins gets wedged again. and finally, the obligatory IT Crowd clip: https://www.youtube.com/watch?v=5UT8RkSmN4k shane (who expects jenkins to crash within 5 minutes of this email going out) On Fri, Mar 15, 2019 at 8:22 PM Sean Owen wrote: > It's not responding again. Is there any way to kick it harder? I know > it's well understood but this means not much can be merged in Spark > > On Fri, Mar 15, 2019 at 12:08 PM shane knapp > wrote: > > > > well, that box rebooted in record time! we're back up and building. > > > > and as always, i'll keep a close eye on things today... jenkins > usually works great, until it doesn't. :\ > > > > On Fri, Mar 15, 2019 at 9:52 AM shane knapp > wrote: > >> > >> as some of you may have noticed, jenkins got itself in a bad state > multiple times over the past couple of weeks. usually restarting the > service is sufficient, but it appears that i need to hit it w/the reboot > hammer. > >> > >> jenkins will be down for the next 20-30 minutes as the node reboots > and jenkins spins back up. i'll reply here w/any updates. > >> > >> shane > >> -- > >> Shane Knapp > >> UC Berkeley EECS Research / RISELab Staff Technical Lead > >> https://rise.cs.berkeley.edu > > > > > > > > -- > > Shane Knapp > > UC Berkeley EECS Research / RISELab Staff Technical Lead > > https://rise.cs.berkeley.edu > -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu >>> >> >> -- >> Shane Knapp >> UC Berkeley EECS Research / RISELab Staff Technical Lead >> https://rise.cs.berkeley.edu >> > > > -- > Shane Knapp > UC Berkeley EECS Research / RISELab Staff Technical Lead > https://rise.cs.berkeley.edu > -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu
Re: Manually reading parquet files.
Try `val enconder = RowEncoder(df.schema).resolveAndBind()` ?
On Thu, Mar 21, 2019 at 5:39 PM Long, Andrew
wrote:
> Thanks a ton for the help!
>
>
>
> Is there a standardized way of converting the internal row to rows?
>
>
>
> I’ve tried this but im getting an exception
>
>
>
> *val *enconder = *RowEncoder*(df.schema)
> *val *rows = readFile(pFile).flatMap(_ *match *{
> *case *r: InternalRow => *Seq*(r)
> *case *b: ColumnarBatch => b.rowIterator().asScala
> })
> .map(enconder.fromRow(_))
> .toList
>
>
>
> java.lang.RuntimeException: Error while decoding:
> java.lang.UnsupportedOperationException: Cannot evaluate expression:
> getcolumnbyordinal(0, IntegerType)
>
> createexternalrow(getcolumnbyordinal(0, IntegerType),
> getcolumnbyordinal(1, IntegerType), getcolumnbyordinal(2,
> StringType).toString, StructField(pk,IntegerType,false),
> StructField(ordering,IntegerType,false), StructField(col_a,StringType,true))
>
>
>
> at
> org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.fromRow(ExpressionEncoder.scala:305)
>
> at
> com.amazon.horizon.azulene.ParquetReadTests$$anonfun$2.apply(ParquetReadTests.scala:100)
>
> at
> com.amazon.horizon.azulene.ParquetReadTests$$anonfun$2.apply(ParquetReadTests.scala:100)
>
>
>
> *From: *Ryan Blue
> *Reply-To: *"rb...@netflix.com"
> *Date: *Thursday, March 21, 2019 at 3:32 PM
> *To: *"Long, Andrew"
> *Cc: *"dev@spark.apache.org" , "
> u...@spark.apache.org" , "horizon-...@amazon.com" <
> horizon-...@amazon.com>
> *Subject: *Re: Manually reading parquet files.
>
>
>
> You're getting InternalRow instances. They probably have the data you
> want, but the toString representation doesn't match the data for
> InternalRow.
>
>
>
> On Thu, Mar 21, 2019 at 3:28 PM Long, Andrew
> wrote:
>
> Hello Friends,
>
>
>
> I’m working on a performance improvement that reads additional parquet
> files in the middle of a lambda and I’m running into some issues. This is
> what id like todo
>
>
>
> ds.mapPartitions(x=>{
> //read parquet file in and perform an operation with x
> })
>
>
>
>
>
> Here’s my current POC code but I’m getting nonsense back from the row
> reader.
>
>
>
> *import *com.amazon.horizon.azulene.util.SparkFileUtils._
>
> *spark*.*conf*.set("spark.sql.parquet.enableVectorizedReader","false")
>
> *val *data = *List*(
> *TestRow*(1,1,"asdf"),
> *TestRow*(2,1,"asdf"),
> *TestRow*(3,1,"asdf"),
> *TestRow*(4,1,"asdf")
> )
>
> *val *df = *spark*.createDataFrame(data)
>
> *val *folder = Files.*createTempDirectory*("azulene-test")
>
> *val *folderPath = folder.toAbsolutePath.toString + "/"
> df.write.mode("overwrite").parquet(folderPath)
>
> *val *files = *spark*.fs.listStatus(folder.toUri)
>
> *val *file = files(1)//skip _success file
>
> *val *partitionSchema = *StructType*(*Seq*.empty)
> *val *dataSchema = df.schema
> *val *fileFormat = *new *ParquetFileFormat()
>
> *val *path = file.getPath
>
> *val *status = *spark*.fs.getFileStatus(path)
>
> *val *pFile = *new *PartitionedFile(
> partitionValues = InternalRow.*empty*,//This should be empty for non
> partitioned values
> filePath = path.toString,
> start = 0,
> length = status.getLen
> )
>
> *val *readFile: (PartitionedFile) => Iterator[Any] =
> //Iterator[InternalRow]
> fileFormat.buildReaderWithPartitionValues(
> sparkSession = *spark*,
> dataSchema = dataSchema,
> partitionSchema = partitionSchema,//this should be empty for non
> partitioned feilds
> requiredSchema = dataSchema,
> filters = *Seq*.empty,
> options = *Map*.*empty*,
> hadoopConf = *spark*.sparkContext.hadoopConfiguration
> //relation.sparkSession.sessionState.newHadoopConfWithOptions(relation.options))
> )
>
> *import *scala.collection.JavaConverters._
>
> *val *rows = readFile(pFile).flatMap(_ *match *{
> *case *r: InternalRow => *Seq*(r)
>
> // This doesn't work. vector mode is doing something screwy
> *case *b: ColumnarBatch => b.rowIterator().asScala
> }).toList
>
> *println*(rows)
> //List([0,1,5b,24,66647361])
> //??this is wrong I think
>
>
>
> Has anyone attempted something similar?
>
>
>
> Cheers Andrew
>
>
>
>
>
>
> --
>
> Ryan Blue
>
> Software Engineer
>
> Netflix
>
Re: spark sql occer error
Did you include the whole error message? On Fri, Mar 22, 2019 at 12:45 AM 563280193 <563280...@qq.com> wrote: > Hi , > I ran a spark sql like this: > > *select imei,tag, product_id,* > * sum(case when succ1>=1 then 1 else 0 end) as succ,* > * sum(case when fail1>=1 and succ1=0 then 1 else 0 end) as fail,* > * count(*) as cnt* > *from t_tbl* > *where sum(case when succ1>=1 then 1 else 0 end)=0 and sum(case when > fail1>=1 and succ1=0 then 1 else 0 end)>0* > *group by **tag, product_id, app_version* > > It occur a problem below: > > * execute, tree:* > *Exchange hashpartitioning(imei#0, tag#1, product_id#2, 100)* > *+- *(1) HashAggregate(keys=[imei#0, tag#1, product_id#2], > functions=[partial_sum(cast(CASE WHEN (succ1#3L >= 1) THEN 1 ELSE 0 END as > bigint)), partial_sum(cast(CASE WHEN ((fail1#4L >= 1) && (succ1#3L = 0)) > THEN 1 ELSE 0 END as bigint)), partial_count(1)], output=[imei#0, tag#1, > product_id#2, sum#49L, sum#50L, count#51L])* > * +- *(1) Filter ((sum(cast(CASE WHEN (succ1#3L >= 1) THEN 1 ELSE 0 END > as bigint)) = 0) && (sum(cast(CASE WHEN ((fail1#4L >= 1) && (succ1#3L = 0)) > THEN 1 ELSE 0 END as bigint)) > 0))* > * +- *(1) FileScan json [imei#0,tag#1,product_id#2,succ1#3L,fail1#4L] > Batched: false, Format: JSON, Location: InMemoryFileIndex[hdfs://xx], > PartitionFilters: [], PushedFilters: [], ReadSchema: > struct* > > > Could anyone help me to solve this problem? > my spark version is 2.3.1 > thank you. >
Re: [build system] jenkins wedged again, rebooting master node
i was right to not hold my breath... while my apache changes seem to have helped a bit, things are still slowing down after 10-12 hours. i have a few other things i can look at, and will get as much done as possible before the weekend. serious troubleshooting will begin anew monday. apologies again, shane On Thu, Mar 21, 2019 at 12:54 PM shane knapp wrote: > i tweaked some apache settings (MaxClients increased to fix an error i > found buried in the logs, and added 'retry' and 'acquire' to the reverse > proxy settings to hopefully combat the dreaded 502 response), restarted > httpd and things actually seem quite snappy right now! > > i'm not holding my breath, however... only time will tell. > > On Tue, Mar 19, 2019 at 7:18 AM Imran Rashid wrote: > >> seems wedged again? >> >> sorry for the bad news Shane, thanks for all the work on fixing it >> >> On Mon, Mar 18, 2019 at 4:02 PM shane knapp wrote: >> >>> ok, i dug through the logs and noticed that rsyslogd was dropping >>> messages to do imuxsock being spammed by postfix... which i then tracked >>> down to our installation of fail2ban being incorrectly configured and >>> attempting to send IP ban/unban status emails to 'em...@example.com'. >>> >>> since we're a university, and especially one w/a reputation like ours, >>> we are constantly under attack. the logs of the attempted dictionary >>> attacks would astound you in their size and scope. since we have so many >>> ban/unban actions happening for all of these unique IP address, each of >>> which generates an email that was directed to an invalid address, we ended >>> up w/well over 100M of plain-text messages waiting in the mail queue. >>> postfix was continually trying to send these messages, which was causing >>> the system to behave strangely, including breaking rsyslogd. >>> >>> so, i disabled email reports in fail2ban, restarted the impacted >>> services, picked my sysadmin's brain and then purged the mail queue (when >>> was the last time anyone actually used postfix?). jenkins now seems to be >>> behaving (maybe?). >>> >>> i'm not entirely sure that this will fix the strange GUI hangs, but all >>> reports i found on stackoverflow and other sites detail strange system >>> behavior across the board when rsyslogd starts dropping messages. at the >>> very least we won't be (potentially) losing system-level log messages >>> anymore, which might actually help me track down what's happening if >>> jenkins gets wedged again. >>> >>> and finally, the obligatory IT Crowd clip: >>> https://www.youtube.com/watch?v=5UT8RkSmN4k >>> >>> shane (who expects jenkins to crash within 5 minutes of this email going >>> out) >>> >>> On Fri, Mar 15, 2019 at 8:22 PM Sean Owen wrote: >>> It's not responding again. Is there any way to kick it harder? I know it's well understood but this means not much can be merged in Spark On Fri, Mar 15, 2019 at 12:08 PM shane knapp wrote: > > well, that box rebooted in record time! we're back up and building. > > and as always, i'll keep a close eye on things today... jenkins usually works great, until it doesn't. :\ > > On Fri, Mar 15, 2019 at 9:52 AM shane knapp wrote: >> >> as some of you may have noticed, jenkins got itself in a bad state multiple times over the past couple of weeks. usually restarting the service is sufficient, but it appears that i need to hit it w/the reboot hammer. >> >> jenkins will be down for the next 20-30 minutes as the node reboots and jenkins spins back up. i'll reply here w/any updates. >> >> shane >> -- >> Shane Knapp >> UC Berkeley EECS Research / RISELab Staff Technical Lead >> https://rise.cs.berkeley.edu > > > > -- > Shane Knapp > UC Berkeley EECS Research / RISELab Staff Technical Lead > https://rise.cs.berkeley.edu >>> >>> >>> -- >>> Shane Knapp >>> UC Berkeley EECS Research / RISELab Staff Technical Lead >>> https://rise.cs.berkeley.edu >>> >> > > -- > Shane Knapp > UC Berkeley EECS Research / RISELab Staff Technical Lead > https://rise.cs.berkeley.edu > -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu
Jenkins issue
Hi All, Seems like there is a jenkins issue again. After a PR builder unit test failure I'm not able to open the jenkins page to take a look at the issue (it got stuck in infinite wait). BR, G
spark sql occer error
Hi , I ran a spark sql like this: select imei,tag, product_id, sum(case when succ1>=1 then 1 else 0 end) as succ, sum(case when fail1>=1 and succ1=0 then 1 else 0 end) as fail, count(*) as cnt from t_tbl where sum(case when succ1>=1 then 1 else 0 end)=0 and sum(case when fail1>=1 and succ1=0 then 1 else 0 end)>0 group by tag, product_id, app_version It occur a problem below: execute, tree: Exchange hashpartitioning(imei#0, tag#1, product_id#2, 100) +- *(1) HashAggregate(keys=[imei#0, tag#1, product_id#2], functions=[partial_sum(cast(CASE WHEN (succ1#3L >= 1) THEN 1 ELSE 0 END as bigint)), partial_sum(cast(CASE WHEN ((fail1#4L >= 1) && (succ1#3L = 0)) THEN 1 ELSE 0 END as bigint)), partial_count(1)], output=[imei#0, tag#1, product_id#2, sum#49L, sum#50L, count#51L]) +- *(1) Filter ((sum(cast(CASE WHEN (succ1#3L >= 1) THEN 1 ELSE 0 END as bigint)) = 0) && (sum(cast(CASE WHEN ((fail1#4L >= 1) && (succ1#3L = 0)) THEN 1 ELSE 0 END as bigint)) > 0)) +- *(1) FileScan json [imei#0,tag#1,product_id#2,succ1#3L,fail1#4L] Batched: false, Format: JSON, Location: InMemoryFileIndex[hdfs://xx], PartitionFilters: [], PushedFilters: [], ReadSchema: struct Could anyone help me to solve this problem? my spark version is 2.3.1 thank you.
spark sql occer error
Hi , I ran a spark sql like this: select imei,tag, product_id, sum(case when succ1>=1 then 1 else 0 end) as succ, sum(case when fail1>=1 and succ1=0 then 1 else 0 end) as fail, count(*) as cnt from t_tbl where sum(case when succ1>=1 then 1 else 0 end)=0 and sum(case when fail1>=1 and succ1=0 then 1 else 0 end)>0 group by tag, product_id, app_version It occur a problem below: execute, tree: Exchange hashpartitioning(imei#0, tag#1, product_id#2, 100) +- *(1) HashAggregate(keys=[imei#0, tag#1, product_id#2], functions=[partial_sum(cast(CASE WHEN (succ1#3L >= 1) THEN 1 ELSE 0 END as bigint)), partial_sum(cast(CASE WHEN ((fail1#4L >= 1) && (succ1#3L = 0)) THEN 1 ELSE 0 END as bigint)), partial_count(1)], output=[imei#0, tag#1, product_id#2, sum#49L, sum#50L, count#51L]) +- *(1) Filter ((sum(cast(CASE WHEN (succ1#3L >= 1) THEN 1 ELSE 0 END as bigint)) = 0) && (sum(cast(CASE WHEN ((fail1#4L >= 1) && (succ1#3L = 0)) THEN 1 ELSE 0 END as bigint)) > 0)) +- *(1) FileScan json [imei#0,tag#1,product_id#2,succ1#3L,fail1#4L] Batched: false, Format: JSON, Location: InMemoryFileIndex[hdfs://xx], PartitionFilters: [], PushedFilters: [], ReadSchema: struct Could anyone help me to solve this problem? my spark version is 2.3.1 thank you. 563280...@qq.com
Re: [VOTE] Release Apache Spark 2.4.1 (RC8)
Since we have couple concerns and hesitations to release rc8, how about we give it couple days, and have another vote on March 25, Monday? In this case, I will cut another rc9 in the Monday morning. Darcy, as Dongjoon mentioned, https://github.com/apache/spark/pull/24092 is conflict against branch-2.4, can you make anther PR against branch-2.4 so we can include the ORC fix in 2.4.1? Thanks. Sincerely, DB Tsai -- Web: https://www.dbtsai.com PGP Key ID: 42E5B25A8F7A82C1 On Wed, Mar 20, 2019 at 9:11 PM Felix Cheung wrote: > > Reposting for shane here > > [SPARK-27178] > https://github.com/apache/spark/commit/342e91fdfa4e6ce5cc3a0da085d1fe723184021b > > Is problematic too and it’s not in the rc8 cut > > https://github.com/apache/spark/commits/branch-2.4 > > (Personally I don’t want to delay 2.4.1 either..) > > > From: Sean Owen > Sent: Wednesday, March 20, 2019 11:18 AM > To: DB Tsai > Cc: dev > Subject: Re: [VOTE] Release Apache Spark 2.4.1 (RC8) > > +1 for this RC. The tag is correct, licenses and sigs check out, tests > of the source with most profiles enabled works for me. > > On Tue, Mar 19, 2019 at 5:28 PM DB Tsai wrote: > > > > Please vote on releasing the following candidate as Apache Spark version > > 2.4.1. > > > > The vote is open until March 23 PST and passes if a majority +1 PMC votes > > are cast, with > > a minimum of 3 +1 votes. > > > > [ ] +1 Release this package as Apache Spark 2.4.1 > > [ ] -1 Do not release this package because ... > > > > To learn more about Apache Spark, please see http://spark.apache.org/ > > > > The tag to be voted on is v2.4.1-rc8 (commit > > 746b3ddee6f7ad3464e326228ea226f5b1f39a41): > > https://github.com/apache/spark/tree/v2.4.1-rc8 > > > > The release files, including signatures, digests, etc. can be found at: > > https://dist.apache.org/repos/dist/dev/spark/v2.4.1-rc8-bin/ > > > > Signatures used for Spark RCs can be found in this file: > > https://dist.apache.org/repos/dist/dev/spark/KEYS > > > > The staging repository for this release can be found at: > > https://repository.apache.org/content/repositories/orgapachespark-1318/ > > > > The documentation corresponding to this release can be found at: > > https://dist.apache.org/repos/dist/dev/spark/v2.4.1-rc8-docs/ > > > > The list of bug fixes going into 2.4.1 can be found at the following URL: > > https://issues.apache.org/jira/projects/SPARK/versions/2.4.1 > > > > FAQ > > > > = > > How can I help test this release? > > = > > > > If you are a Spark user, you can help us test this release by taking > > an existing Spark workload and running on this release candidate, then > > reporting any regressions. > > > > If you're working in PySpark you can set up a virtual env and install > > the current RC and see if anything important breaks, in the Java/Scala > > you can add the staging repository to your projects resolvers and test > > with the RC (make sure to clean up the artifact cache before/after so > > you don't end up building with a out of date RC going forward). > > > > === > > What should happen to JIRA tickets still targeting 2.4.1? > > === > > > > The current list of open tickets targeted at 2.4.1 can be found at: > > https://issues.apache.org/jira/projects/SPARK and search for "Target > > Version/s" = 2.4.1 > > > > Committers should look at those and triage. Extremely important bug > > fixes, documentation, and API tweaks that impact compatibility should > > be worked on immediately. Everything else please retarget to an > > appropriate release. > > > > == > > But my bug isn't fixed? > > == > > > > In order to make timely releases, we will typically not hold the > > release unless the bug in question is a regression from the previous > > release. That being said, if there is something which is a regression > > that has not been correctly targeted please ping me or a committer to > > help target the issue. > > > > > > DB Tsai | Siri Open Source Technologies [not a contribution] | Apple, Inc > > > > > > - > > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org > > > > - > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org > - To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
