[GitHub] any23 pull request #42: ANY23-308 [NEW] Adding option -d support for yaml fi...
Github user asfgit closed the pull request at: https://github.com/apache/any23/pull/42 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] any23 pull request #42: ANY23-308 [NEW] Adding option -d support for yaml fi...
Github user jgrzebyta commented on a diff in the pull request:
https://github.com/apache/any23/pull/42#discussion_r127026645
--- Diff:
utils/src/main/java/org/apache/any23/extractor/yaml/YAMLValidator.java ---
@@ -0,0 +1,105 @@
+/*
+ * Copyright 2017 The Apache Software Foundation.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.any23.extractor.yaml;
+
+import com.google.common.collect.Iterables;
+import java.io.BufferedInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.Scanner;
+import java.util.regex.Pattern;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.yaml.snakeyaml.Yaml;
+
+/**
+ * Utility class provides static methods for YAML validation.
+ *
+ * @author Jacek Grzebyta (grzebyta.dev [at] gmail.com)
+ */
+public class YAMLValidator {
+
+private static final Logger log =
LoggerFactory.getLogger(YAMLValidator.class);
+
+private static final Pattern YAML_PATTERN =
Pattern.compile("^%YAML.*", Pattern.CASE_INSENSITIVE);
+
+/**
+ * Detects if is contains valid YAML content.
+ *
+ * In the first instance it checks if there is "%YAML" head. If not
check
+ * using the brute force method by parsing input stream with yaml
parser.
+ *
+ *
+ * NB. Only "false" results are trusted. Even if result is "true" you
cannot
+ * be sure that InputStream contains YAML intentional context because
+ * comma-separated-values are pars-able by YAML parser as well.
+ *
+ *
+ * @param is {@link InputStream}
+ * @return
+ * @throws IOException
+ */
+public static boolean isYAML(InputStream is) throws IOException {
+if (is == null) {
+return false;
+}
+
+if (!is.markSupported()) {
--- End diff --
I have checked in the source and `markSupported()` method olny contains
`return true` or `return false`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] any23 pull request #42: ANY23-308 [NEW] Adding option -d support for yaml fi...
Github user jgrzebyta commented on a diff in the pull request:
https://github.com/apache/any23/pull/42#discussion_r127007051
--- Diff:
core/src/test/java/org/apache/any23/extractor/yaml/YAMLExtractorTest.java ---
@@ -93,4 +92,17 @@ public void nullTest()
RepositoryResult docs = getStatements(null, null,
RDF.NIL);
Assert.assertTrue(Iterations.asList(docs).size() == 2);
}
+
+/**
+ * Comma separated values are parsed as well.
+ *
+ * @throws Exception
+ */
+@Test
+public void csvTest()
+throws Exception {
+assertExtract("/org/apache/any23/extractor/csv/test-comma.csv");
--- End diff --
That is false positive. It is just proof that for text-based formats
additional recognising should be done from `meta`. I will delete that.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] any23 pull request #42: ANY23-308 [NEW] Adding option -d support for yaml fi...
Github user ansell commented on a diff in the pull request:
https://github.com/apache/any23/pull/42#discussion_r126829654
--- Diff:
utils/src/main/java/org/apache/any23/extractor/yaml/YAMLValidator.java ---
@@ -0,0 +1,105 @@
+/*
+ * Copyright 2017 The Apache Software Foundation.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.any23.extractor.yaml;
+
+import com.google.common.collect.Iterables;
+import java.io.BufferedInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.Scanner;
+import java.util.regex.Pattern;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.yaml.snakeyaml.Yaml;
+
+/**
+ * Utility class provides static methods for YAML validation.
+ *
+ * @author Jacek Grzebyta (grzebyta.dev [at] gmail.com)
+ */
+public class YAMLValidator {
+
+private static final Logger log =
LoggerFactory.getLogger(YAMLValidator.class);
+
+private static final Pattern YAML_PATTERN =
Pattern.compile("^%YAML.*", Pattern.CASE_INSENSITIVE);
+
+/**
+ * Detects if is contains valid YAML content.
+ *
+ * In the first instance it checks if there is "%YAML" head. If not
check
+ * using the brute force method by parsing input stream with yaml
parser.
+ *
+ *
+ * NB. Only "false" results are trusted. Even if result is "true" you
cannot
+ * be sure that InputStream contains YAML intentional context because
+ * comma-separated-values are pars-able by YAML parser as well.
+ *
+ *
+ * @param is {@link InputStream}
+ * @return
+ * @throws IOException
+ */
+public static boolean isYAML(InputStream is) throws IOException {
+if (is == null) {
+return false;
+}
+
+if (!is.markSupported()) {
+is = new BufferedInputStream(is);
+}
+
+boolean result = false;
+
+// mark the reading frame position. MUST BE FIRST
+is.mark(Integer.MAX_VALUE);
+
+while (true) {
+// if is is empty than return false
+if (is.available() <= 0) {
+break;
+}
+
+Scanner sc = new Scanner(is);
+String out = sc.findWithinHorizon(YAML_PATTERN, 0);
+
+if (out != null && !out.isEmpty()) {
+log.debug("Head: {}", out);
+result = true;
+break;
+}
+log.debug("Still not found. output is: {}", out);
+is.reset();
+
+try {
+Yaml yml = new Yaml();
+Iterable parsedOut = yml.loadAll(is);
+
+if (Iterables.size(parsedOut) > 0) {
+result = true;
+break;
+}
+} catch (Exception ex) {
+//do nothing
--- End diff --
This should return false rather than silently aborting
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] any23 pull request #42: ANY23-308 [NEW] Adding option -d support for yaml fi...
Github user ansell commented on a diff in the pull request:
https://github.com/apache/any23/pull/42#discussion_r126829870
--- Diff:
utils/src/main/java/org/apache/any23/extractor/yaml/YAMLValidator.java ---
@@ -0,0 +1,105 @@
+/*
+ * Copyright 2017 The Apache Software Foundation.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.any23.extractor.yaml;
+
+import com.google.common.collect.Iterables;
+import java.io.BufferedInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.Scanner;
+import java.util.regex.Pattern;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.yaml.snakeyaml.Yaml;
+
+/**
+ * Utility class provides static methods for YAML validation.
+ *
+ * @author Jacek Grzebyta (grzebyta.dev [at] gmail.com)
+ */
+public class YAMLValidator {
+
+private static final Logger log =
LoggerFactory.getLogger(YAMLValidator.class);
+
+private static final Pattern YAML_PATTERN =
Pattern.compile("^%YAML.*", Pattern.CASE_INSENSITIVE);
+
+/**
+ * Detects if is contains valid YAML content.
+ *
+ * In the first instance it checks if there is "%YAML" head. If not
check
+ * using the brute force method by parsing input stream with yaml
parser.
+ *
+ *
+ * NB. Only "false" results are trusted. Even if result is "true" you
cannot
+ * be sure that InputStream contains YAML intentional context because
+ * comma-separated-values are pars-able by YAML parser as well.
+ *
+ *
+ * @param is {@link InputStream}
+ * @return
+ * @throws IOException
+ */
+public static boolean isYAML(InputStream is) throws IOException {
+if (is == null) {
+return false;
+}
+
+if (!is.markSupported()) {
+is = new BufferedInputStream(is);
+}
+
+boolean result = false;
+
+// mark the reading frame position. MUST BE FIRST
+is.mark(Integer.MAX_VALUE);
+
+while (true) {
--- End diff --
This loop is not being used. Replace it with try-finally and replace the
"result =..." cases with "return ...".
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] any23 pull request #42: ANY23-308 [NEW] Adding option -d support for yaml fi...
Github user ansell commented on a diff in the pull request:
https://github.com/apache/any23/pull/42#discussion_r126829615
--- Diff:
utils/src/main/java/org/apache/any23/extractor/yaml/YAMLValidator.java ---
@@ -0,0 +1,105 @@
+/*
+ * Copyright 2017 The Apache Software Foundation.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.any23.extractor.yaml;
+
+import com.google.common.collect.Iterables;
+import java.io.BufferedInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.Scanner;
+import java.util.regex.Pattern;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.yaml.snakeyaml.Yaml;
+
+/**
+ * Utility class provides static methods for YAML validation.
+ *
+ * @author Jacek Grzebyta (grzebyta.dev [at] gmail.com)
+ */
+public class YAMLValidator {
+
+private static final Logger log =
LoggerFactory.getLogger(YAMLValidator.class);
+
+private static final Pattern YAML_PATTERN =
Pattern.compile("^%YAML.*", Pattern.CASE_INSENSITIVE);
+
+/**
+ * Detects if is contains valid YAML content.
+ *
+ * In the first instance it checks if there is "%YAML" head. If not
check
+ * using the brute force method by parsing input stream with yaml
parser.
+ *
+ *
+ * NB. Only "false" results are trusted. Even if result is "true" you
cannot
+ * be sure that InputStream contains YAML intentional context because
+ * comma-separated-values are pars-able by YAML parser as well.
+ *
+ *
+ * @param is {@link InputStream}
+ * @return
+ * @throws IOException
+ */
+public static boolean isYAML(InputStream is) throws IOException {
+if (is == null) {
+return false;
+}
+
+if (!is.markSupported()) {
+is = new BufferedInputStream(is);
+}
+
+boolean result = false;
+
+// mark the reading frame position. MUST BE FIRST
+is.mark(Integer.MAX_VALUE);
+
+while (true) {
+// if is is empty than return false
+if (is.available() <= 0) {
+break;
+}
+
+Scanner sc = new Scanner(is);
+String out = sc.findWithinHorizon(YAML_PATTERN, 0);
+
+if (out != null && !out.isEmpty()) {
+log.debug("Head: {}", out);
+result = true;
+break;
+}
+log.debug("Still not found. output is: {}", out);
+is.reset();
+
+try {
+Yaml yml = new Yaml();
+Iterable parsedOut = yml.loadAll(is);
+
+if (Iterables.size(parsedOut) > 0) {
+result = true;
+break;
+}
+} catch (Exception ex) {
+//do nothing
+}
+
+// final break
+break;
--- End diff --
breaking at the end of a while(true) loop with no calls to "continue" shows
the code needs to be changed. There doesn't seem to be anything iterative about
the algorithm, so not clear why you are doing it this way.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
