Re: Attempting to accept an unknown offer
That error message occurs when the Mesos scheduler tries to accept an offer that doesn't exist. It should never happen. Can you submit a JIRA and cc me to it? Also, what libmesos and mesos master version are you running? On Wed, Aug 17, 2016 at 9:23 AM, vr sparkwrote: > My code is very simple, if i use other hive tables, my code works fine. This > particular table (virtual view) is huge and might have more metadata. > > It has only two columns. > > virtual view name is : cluster_table > > # col_namedata_type > > ln string > > parti int > > > here is snippet... > > from pyspark import SparkConf, SparkContext > > from pyspark.sql import HiveContext > > import pyspark.sql > > import json > > myconf=SparkConf().setAppName("sql") > > spcont=SparkContext(conf=myconf) > > sqlcont=HiveContext(spcont) > > res=sqlcont.sql("select parti FROM h.cluster_table WHERE parti > 408910 > and parti <408911 limit 10") > > print res.printSchema() > > print 'res' > > print res > > df=res.collect() > > print 'after collect' > > print df > > > Here is the ouput after i submit the job > > I0817 09:18:40.606465 31409 sched.cpp:262] New master detected at > master@x.y.17.56:6750 > > I0817 09:18:40.607461 31409 sched.cpp:272] No credentials provided. > Attempting to register without authentication > > I0817 09:18:40.612763 31409 sched.cpp:641] Framework registered with > b859f2f3-7484-482d-8c0d-35bd91c1ad0a-6336 > > 16/08/17 09:18:57 WARN ObjectStore: Version information not found in > metastore. hive.metastore.schema.verification is not enabled so recording > the schema version 1.2.0 > > 16/08/17 09:18:57 WARN ObjectStore: Failed to get database default, > returning NoSuchObjectException > > root > > |-- parti: integer (nullable = true) > > > None > > res > > DataFrame[partition_epoch_hourtenth: int] > > 2016-08-17 09:19:20,648:31315(0x7fafebfb1700):ZOO_WARN@ > zookeeper_interest@1557: Exceeded deadline by 19ms > > 2016-08-17 09:19:30,662:31315(0x7fafebfb1700):ZOO_WARN@ > zookeeper_interest@1557: Exceeded deadline by 13ms > > W0817 09:20:01.715824 31412 sched.cpp:1195] Attempting to accept an > unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O168676564 > > W0817 09:20:01.716455 31412 sched.cpp:1195] Attempting to accept an > unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O168676630 > > W0817 09:20:01.716645 31412 sched.cpp:1195] Attempting to accept an > unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O168676713 > > W0817 09:20:01.724409 31412 sched.cpp:1195] Attempting to accept an > unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O168676554 > > W0817 09:20:01.724728 31412 sched.cpp:1195] Attempting to accept an > unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O168676555 > > W0817 09:20:01.724936 31412 sched.cpp:1195] Attempting to accept an > unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O168676556 > > W0817 09:20:01.725126 31412 sched.cpp:1195] Attempting to accept an > unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O168676557 > > W0817 09:20:01.725309 31412 sched.cpp:1195] Attempting to accept an > unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O168676558. > > and many more lines like this on the screen with similar message > > > > On Wed, Aug 17, 2016 at 9:08 AM, Ted Yu wrote: > >> Please include user@ in your reply. >> >> Can you reveal the snippet of hive sql ? >> >> On Wed, Aug 17, 2016 at 9:04 AM, vr spark wrote: >> >>> spark 1.6.1 >>> mesos >>> job is running for like 10-15 minutes and giving this message and i >>> killed it. >>> >>> In this job, i am creating data frame from a hive sql. There are other >>> similar jobs which work fine >>> >>> On Wed, Aug 17, 2016 at 8:52 AM, Ted Yu wrote: >>> Can you provide more information ? Were you running on YARN ? Which version of Spark are you using ? Was your job failing ? Thanks On Wed, Aug 17, 2016 at 8:46 AM, vr spark wrote: > > W0816 23:17:01.984846 16360 sched.cpp:1195] Attempting to accept an > unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910492 > > W0816 23:17:01.984987 16360 sched.cpp:1195] Attempting to accept an > unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910493 > > W0816 23:17:01.985124 16360 sched.cpp:1195] Attempting to accept an > unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910494 > > W0816 23:17:01.985339 16360 sched.cpp:1195] Attempting to accept an > unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910495 > > W0816 23:17:01.985508 16360 sched.cpp:1195] Attempting to accept an > unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910496 > > W0816 23:17:01.985651 16360 sched.cpp:1195] Attempting to accept an > unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910497 > > W0816
Re: Attempting to accept an unknown offer
My code is very simple, if i use other hive tables, my code works fine. This particular table (virtual view) is huge and might have more metadata. It has only two columns. virtual view name is : cluster_table # col_namedata_type ln string parti int here is snippet... from pyspark import SparkConf, SparkContext from pyspark.sql import HiveContext import pyspark.sql import json myconf=SparkConf().setAppName("sql") spcont=SparkContext(conf=myconf) sqlcont=HiveContext(spcont) res=sqlcont.sql("select parti FROM h.cluster_table WHERE parti > 408910 and parti <408911 limit 10") print res.printSchema() print 'res' print res df=res.collect() print 'after collect' print df Here is the ouput after i submit the job I0817 09:18:40.606465 31409 sched.cpp:262] New master detected at master@x.y.17.56:6750 I0817 09:18:40.607461 31409 sched.cpp:272] No credentials provided. Attempting to register without authentication I0817 09:18:40.612763 31409 sched.cpp:641] Framework registered with b859f2f3-7484-482d-8c0d-35bd91c1ad0a-6336 16/08/17 09:18:57 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.2.0 16/08/17 09:18:57 WARN ObjectStore: Failed to get database default, returning NoSuchObjectException root |-- parti: integer (nullable = true) None res DataFrame[partition_epoch_hourtenth: int] 2016-08-17 09:19:20,648:31315(0x7fafebfb1700):ZOO_WARN@zookeeper_interest@1557: Exceeded deadline by 19ms 2016-08-17 09:19:30,662:31315(0x7fafebfb1700):ZOO_WARN@zookeeper_interest@1557: Exceeded deadline by 13ms W0817 09:20:01.715824 31412 sched.cpp:1195] Attempting to accept an unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O168676564 W0817 09:20:01.716455 31412 sched.cpp:1195] Attempting to accept an unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O168676630 W0817 09:20:01.716645 31412 sched.cpp:1195] Attempting to accept an unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O168676713 W0817 09:20:01.724409 31412 sched.cpp:1195] Attempting to accept an unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O168676554 W0817 09:20:01.724728 31412 sched.cpp:1195] Attempting to accept an unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O168676555 W0817 09:20:01.724936 31412 sched.cpp:1195] Attempting to accept an unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O168676556 W0817 09:20:01.725126 31412 sched.cpp:1195] Attempting to accept an unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O168676557 W0817 09:20:01.725309 31412 sched.cpp:1195] Attempting to accept an unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O168676558. and many more lines like this on the screen with similar message On Wed, Aug 17, 2016 at 9:08 AM, Ted Yuwrote: > Please include user@ in your reply. > > Can you reveal the snippet of hive sql ? > > On Wed, Aug 17, 2016 at 9:04 AM, vr spark wrote: > >> spark 1.6.1 >> mesos >> job is running for like 10-15 minutes and giving this message and i >> killed it. >> >> In this job, i am creating data frame from a hive sql. There are other >> similar jobs which work fine >> >> On Wed, Aug 17, 2016 at 8:52 AM, Ted Yu wrote: >> >>> Can you provide more information ? >>> >>> Were you running on YARN ? >>> Which version of Spark are you using ? >>> >>> Was your job failing ? >>> >>> Thanks >>> >>> On Wed, Aug 17, 2016 at 8:46 AM, vr spark wrote: >>> W0816 23:17:01.984846 16360 sched.cpp:1195] Attempting to accept an unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910492 W0816 23:17:01.984987 16360 sched.cpp:1195] Attempting to accept an unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910493 W0816 23:17:01.985124 16360 sched.cpp:1195] Attempting to accept an unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910494 W0816 23:17:01.985339 16360 sched.cpp:1195] Attempting to accept an unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910495 W0816 23:17:01.985508 16360 sched.cpp:1195] Attempting to accept an unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910496 W0816 23:17:01.985651 16360 sched.cpp:1195] Attempting to accept an unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910497 W0816 23:17:01.985801 16360 sched.cpp:1195] Attempting to accept an unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910498 W0816 23:17:01.985961 16360 sched.cpp:1195] Attempting to accept an unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910499 W0816 23:17:01.986121 16360 sched.cpp:1195] Attempting to accept an unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910500 2016-08-16 23:18:41,877:16226(0x7f71271b6700):ZOO_WARN@zookeeper_intere st@1557: Exceeded
Re: Attempting to accept an unknown offer
Please include user@ in your reply. Can you reveal the snippet of hive sql ? On Wed, Aug 17, 2016 at 9:04 AM, vr sparkwrote: > spark 1.6.1 > mesos > job is running for like 10-15 minutes and giving this message and i killed > it. > > In this job, i am creating data frame from a hive sql. There are other > similar jobs which work fine > > On Wed, Aug 17, 2016 at 8:52 AM, Ted Yu wrote: > >> Can you provide more information ? >> >> Were you running on YARN ? >> Which version of Spark are you using ? >> >> Was your job failing ? >> >> Thanks >> >> On Wed, Aug 17, 2016 at 8:46 AM, vr spark wrote: >> >>> >>> W0816 23:17:01.984846 16360 sched.cpp:1195] Attempting to accept an >>> unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910492 >>> >>> W0816 23:17:01.984987 16360 sched.cpp:1195] Attempting to accept an >>> unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910493 >>> >>> W0816 23:17:01.985124 16360 sched.cpp:1195] Attempting to accept an >>> unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910494 >>> >>> W0816 23:17:01.985339 16360 sched.cpp:1195] Attempting to accept an >>> unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910495 >>> >>> W0816 23:17:01.985508 16360 sched.cpp:1195] Attempting to accept an >>> unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910496 >>> >>> W0816 23:17:01.985651 16360 sched.cpp:1195] Attempting to accept an >>> unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910497 >>> >>> W0816 23:17:01.985801 16360 sched.cpp:1195] Attempting to accept an >>> unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910498 >>> >>> W0816 23:17:01.985961 16360 sched.cpp:1195] Attempting to accept an >>> unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910499 >>> >>> W0816 23:17:01.986121 16360 sched.cpp:1195] Attempting to accept an >>> unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910500 >>> >>> 2016-08-16 23:18:41,877:16226(0x7f71271b6700):ZOO_WARN@zookeeper_intere >>> st@1557: Exceeded deadline by 13ms >>> >>> 2016-08-16 23:21:12,007:16226(0x7f71271b6700):ZOO_WARN@zookeeper_intere >>> st@1557: Exceeded deadline by 11ms >>> >>> >>> >>> >> >
Re: Attempting to accept an unknown offer
Can you provide more information ? Were you running on YARN ? Which version of Spark are you using ? Was your job failing ? Thanks On Wed, Aug 17, 2016 at 8:46 AM, vr sparkwrote: > > W0816 23:17:01.984846 16360 sched.cpp:1195] Attempting to accept an > unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910492 > > W0816 23:17:01.984987 16360 sched.cpp:1195] Attempting to accept an > unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910493 > > W0816 23:17:01.985124 16360 sched.cpp:1195] Attempting to accept an > unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910494 > > W0816 23:17:01.985339 16360 sched.cpp:1195] Attempting to accept an > unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910495 > > W0816 23:17:01.985508 16360 sched.cpp:1195] Attempting to accept an > unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910496 > > W0816 23:17:01.985651 16360 sched.cpp:1195] Attempting to accept an > unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910497 > > W0816 23:17:01.985801 16360 sched.cpp:1195] Attempting to accept an > unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910498 > > W0816 23:17:01.985961 16360 sched.cpp:1195] Attempting to accept an > unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910499 > > W0816 23:17:01.986121 16360 sched.cpp:1195] Attempting to accept an > unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910500 > > 2016-08-16 23:18:41,877:16226(0x7f71271b6700):ZOO_WARN@zookeeper_ > interest@1557: Exceeded deadline by 13ms > > 2016-08-16 23:21:12,007:16226(0x7f71271b6700):ZOO_WARN@zookeeper_ > interest@1557: Exceeded deadline by 11ms > > > >