Re: the scheme in stream reader
Hi , I have submitted a JIRA for this issue. The link is https://issues.apache.org/jira/browse/SPARK-21147 thanks Fei Shao ---Original--- From: "Michael Armbrust"<mich...@databricks.com> Date: 2017/6/20 03:06:49 To: "??"<1427357...@qq.com>; Cc: "user"<user@spark.apache.org>;"dev"<d...@spark.apache.org>; Subject: Re: the scheme in stream reader The socket source can't know how to parse your data. I think the right thing would be for it to throw an exception saying that you can't set the schema here. Would you mind opening a JIRA ticket? If you are trying to parse data from something like JSON then you should use from_json` on the value returned. On Sun, Jun 18, 2017 at 12:27 AM, ?? <1427357...@qq.com> wrote: Hi all, L set the scheme for DataStreamReader but when I print the scheme.It just printed: root |--value:string (nullable=true) My code is val line = ss.readStream.format("socket") .option("ip",xxx) .option("port",xxx) .scheme(StructField("name",StringType??::(StructField("age", IntegerType))).load line.printSchema My spark version is 2.1.0. I want the printSchema prints the schema I set in the code.How should I do please? And my original target is the received data from socket is handled as schema directly.What should I do please? thanks Fei Shao
Re: the scheme in stream reader
The socket source can't know how to parse your data. I think the right thing would be for it to throw an exception saying that you can't set the schema here. Would you mind opening a JIRA ticket? If you are trying to parse data from something like JSON then you should use from_json` on the value returned. On Sun, Jun 18, 2017 at 12:27 AM, 萝卜丝炒饭 <1427357...@qq.com> wrote: > Hi all, > > L set the scheme for DataStreamReader but when I print the scheme.It just > printed: > root > |--value:string (nullable=true) > > My code is > > val line = ss.readStream.format("socket") > .option("ip",xxx) > .option("port",xxx) > .scheme(StructField("name",StringType)::(StructField("age", > IntegerType))).load > line.printSchema > > My spark version is 2.1.0. > I want the printSchema prints the schema I set in the code.How should I do > please? > And my original target is the received data from socket is handled as > schema directly.What should I do please? > > thanks > Fei Shao > > > > > > >
the scheme in stream reader
Hi all, L set the scheme for DataStreamReader but when I print the scheme.It just printed: root |--value:string (nullable=true) My code is val line = ss.readStream.format("socket") .option("ip",xxx) .option("port",xxx) .scheme(StructField("name",StringType??::(StructField("age", IntegerType))).load line.printSchema My spark version is 2.1.0. I want the printSchema prints the schema I set in the code.How should I do please? And my original target is the received data from socket is handled as schema directly.What should I do please? thanks Fei Shao