Hi, Here is an update patch that rebases over the commenting out of pcb_test_window, polygon_triangulation and polygon_generator and fixes a link error to do with base_screen.cpp.
Cheers, John On Mon, Oct 8, 2018 at 5:27 PM John Beard <john.j.be...@gmail.com> wrote: > > Sorry, > > I wrote "ms", I meant "us" - the times are in the handful-of-millsecond range. > > Cheers, > > John > On Mon, Oct 8, 2018 at 5:24 PM John Beard <john.j.be...@gmail.com> wrote: > > > > Hi, > > > > This is a patch to add a test program that allows to parse a Pcbnew > > file from command line params or stdin. This means you can use it for > > fuzz testing. > > > > I have done a little bit of fuzz testing so far (8 million execs, > > about 70% of a cycle), and have not found any crashes, but I can make > > it hang in a few ways. These all seem to be in streams which contain > > nul's. This is actually not reachable from the UI due to reading files > > into wxStrings first (nut quite sure why), whereas this program uses > > the parser directly. Thus, the bug is probably not very critical. > > Example hanging input attached (note there's a nul in it, so your > > editor may or may not like that). > > > > This program can also be fed a number of files, which means it could > > be used for automated testing that all files in a batch can be parsed > > successfully, and also provides a handy way to put GDB on a program > > when debugging the parser against specific input. > > > > There is timing on the parsing too, mostly for interest (use the -v > > flag). It takes about 150-3000ms per FP on my machine for the FPs in > > Connector_PinSocket_2.54mm.pretty. > > > > There's also some centralisation of some QA-related utils into a > > qa_utils library. > > > > Cheers, > > > > John
From 3ff4ee626860c81e48896a1b15ecc967d74e5e39 Mon Sep 17 00:00:00 2001 From: John Beard <john.j.be...@gmail.com> Date: Sat, 6 Oct 2018 15:35:17 +0100 Subject: [PATCH] QA: PCB file input parse test program (fuzzable) This adds a test program which can be used to test the parsing of a given KiCad PCB file. This interface is useful for both manual or automated debugging of given files, as well as providing an interface suitable for fuzz-testing tools. Also adds to the testing docs to detail how fuzzing can be used. Also moves some useful re-usable code from io-benchmark to a new library qa_utils, which can contain code that isn't need in the actual KiCad libs. --- Documentation/development/testing.md | 47 +++++- qa/CMakeLists.txt | 4 + qa/pcb_parse_input/CMakeLists.txt | 66 ++++++++ qa/pcb_parse_input/main.cpp | 158 ++++++++++++++++++ qa/qa_utils/CMakeLists.txt | 40 +++++ qa/qa_utils/scoped_timer.h | 62 +++++++ .../qa_utils}/stdstream_line_reader.cpp | 8 +- .../qa_utils}/stdstream_line_reader.h | 8 +- tools/io_benchmark/CMakeLists.txt | 2 +- 9 files changed, 386 insertions(+), 9 deletions(-) create mode 100644 qa/pcb_parse_input/CMakeLists.txt create mode 100644 qa/pcb_parse_input/main.cpp create mode 100644 qa/qa_utils/CMakeLists.txt create mode 100644 qa/qa_utils/scoped_timer.h rename {tools/io_benchmark => qa/qa_utils}/stdstream_line_reader.cpp (91%) rename {tools/io_benchmark => qa/qa_utils}/stdstream_line_reader.h (92%) diff --git a/Documentation/development/testing.md b/Documentation/development/testing.md index 4726ca536..cf0d4c129 100644 --- a/Documentation/development/testing.md +++ b/Documentation/development/testing.md @@ -103,6 +103,51 @@ You can run the tests in GDB to trace this: If the test segfaults, you will get a familiar backtrace, just like if you were running pcbnew under GDB. +## Fuzz testing ## + +It is possible to run fuzz testing on some parts of KiCad. To do this for a +generic function, you need to be able to pass some kind of input from the fuzz +testing tool to the function under test. + +For example, to use the [AFL fuzzing tool][], you will need: + +* A test executable that can: +** Receive input from `stdin` to be run by `afl-fuzz`. +** Optional: process input from a filename to allow `afl-tmin` to minimise the + input files. +* To compile this executable with an AFL compiler, to enable the instrumentation + that allows the fuzzer to detect the fuzzing state. + +For example, the `qa_pcb_parse_input` executable can be compiled like this: + + mkdir build + cd build + cmake -DCMAKE_CXX_COMPILER=/usr/bin/afl-clang-fast++ -DCMAKE_C_COMPILER=/usr/bin/afl-clang-fast ../kicad_src + make qa_pcb_parse_input + +You may need to disable core dumps and CPU frequency scaling on your system (AFL +will warn you if you should do this). For example, as root: + + # echo core >/proc/sys/kernel/core_pattern + # echo performance | tee cpu*/cpufreq/scaling_governor + +To fuzz: + + afl-fuzz -i fuzzin -o fuzzout -m500 qa/pcb_parse_input/qa_pcb_parse_input + +where: + +* `-i` is a directory of files to use as fuzz input "seeds" +* `-o` is a directory to write the results (including inputs that provoke crashes + or hangs) +* `-t` is the maximum time that a run is allowed to take before being declared a "hang" +* `-m` is the memory allowed to use (this often needs to be bumped, as KiCad code + tends to use a lot of memory to initialise) + +The AFL TUI will then display the fuzzing progress, and you can use the hang- or +crash-provoking inputs to debug code as needed. + [CTest]: https://cmake.org/cmake/help/latest/module/CTest.html [Boost Unit Test framework]: https://www.boost.org/doc/libs/1_68_0/libs/test/doc/html/index.html -[boost-test-functions]: https://www.boost.org/doc/libs/1_68_0/libs/test/doc/html/boost_test/utf_reference/testing_tool_ref.html \ No newline at end of file +[boost-test-functions]: https://www.boost.org/doc/libs/1_68_0/libs/test/doc/html/boost_test/utf_reference/testing_tool_ref.html +[AFL fuzzing tool]: http://lcamtuf.coredump.cx/afl/ \ No newline at end of file diff --git a/qa/CMakeLists.txt b/qa/CMakeLists.txt index 36aa7b601..ae30e1a38 100644 --- a/qa/CMakeLists.txt +++ b/qa/CMakeLists.txt @@ -12,8 +12,12 @@ if( KICAD_SCRIPTING_MODULES ) endif() +# common QA helpers +add_subdirectory( qa_utils ) + add_subdirectory( common ) add_subdirectory( shape_poly_set_refactor ) +add_subdirectory( pcb_parse_input ) # add_subdirectory( pcb_test_window ) # add_subdirectory( polygon_triangulation ) # add_subdirectory( polygon_generator ) \ No newline at end of file diff --git a/qa/pcb_parse_input/CMakeLists.txt b/qa/pcb_parse_input/CMakeLists.txt new file mode 100644 index 000000000..5f8e2f350 --- /dev/null +++ b/qa/pcb_parse_input/CMakeLists.txt @@ -0,0 +1,66 @@ +# This program source code file is part of KiCad, a free EDA CAD application. +# +# Copyright (C) 2018 KiCad Developers, see CHANGELOG.TXT for contributors. +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License +# as published by the Free Software Foundation; either version 2 +# of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, you may find one here: +# http://www.gnu.org/licenses/old-licenses/gpl-2.0.html +# or you may search the http://www.gnu.org website for the version 2 license, +# or you may write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA + + +if( BUILD_GITHUB_PLUGIN ) + set( GITHUB_PLUGIN_LIBRARIES github_plugin ) +endif() + +add_executable( qa_pcb_parse_input + # This is needed for the global mock objects + ../qa_utils/mocks.cpp + + main.cpp + + ../../common/base_units.cpp + ../../common/xnode.cpp + ../../common/base_screen.cpp +) + +include_directories( + ${CMAKE_SOURCE_DIR} + ${CMAKE_SOURCE_DIR}/include + ${CMAKE_SOURCE_DIR}/polygon + ${CMAKE_SOURCE_DIR}/pcbnew + ${CMAKE_SOURCE_DIR}/common + ${CMAKE_SOURCE_DIR}/pcbnew/router + ${CMAKE_SOURCE_DIR}/pcbnew/tools + ${CMAKE_SOURCE_DIR}/pcbnew/dialogs + ${Boost_INCLUDE_DIR} + ${INC_AFTER} +) + +target_link_libraries( qa_pcb_parse_input + legacy_wx + pcbcommon + common + bitmaps + polygon + pcad2kicadpcb + ${GITHUB_PLUGIN_LIBRARIES} + qa_utils + ${wxWidgets_LIBRARIES} +) + +# we need to pretend to be something to appease the units code +target_compile_definitions( qa_pcb_parse_input + PRIVATE PCBNEW +) \ No newline at end of file diff --git a/qa/pcb_parse_input/main.cpp b/qa/pcb_parse_input/main.cpp new file mode 100644 index 000000000..fc79f932b --- /dev/null +++ b/qa/pcb_parse_input/main.cpp @@ -0,0 +1,158 @@ +/* + * This program source code file is part of KiCad, a free EDA CAD application. + * + * Copyright (C) 2018 KiCad Developers, see CHANGELOG.TXT for contributors. + * + * This program is free software; you can redistribute it and/or + * modify it under the terms of the GNU General Public License + * as published by the Free Software Foundation; either version 2 + * of the License, or (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, you may find one here: + * http://www.gnu.org/licenses/old-licenses/gpl-2.0.html + * or you may search the http://www.gnu.org website for the version 2 license, + * or you may write to the Free Software Foundation, Inc., + * 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA + */ + + +#include <kicad_plugin.h> +#include <pcb_parser.h> +#include <richio.h> +#include <class_board_item.h> + +#include <wx/cmdline.h> + +#include <stdstream_line_reader.h> +#include <scoped_timer.h> + +using PARSE_DURATION = std::chrono::microseconds; + +/** + * Parse a PCB or footprint file from the given input stream + * + * @param aStream the input stream to read from + * @return success, duration (in us) + */ +bool parse(std::istream& aStream, bool aVerbose ) +{ + // Take input from stdin + STDISTREAM_LINE_READER reader; + reader.SetStream( aStream ); + + PCB_PARSER parser; + + parser.SetLineReader( &reader ); + + BOARD_ITEM* board = nullptr; + + PARSE_DURATION duration {}; + + try + { + SCOPED_TIMER<PARSE_DURATION> timer( duration ); + board = parser.Parse(); + } + catch( const IO_ERROR& parse_error ) + { + std::cerr << parse_error.Problem() << std::endl; + std::cerr << parse_error.Where() << std::endl; + } + + if( aVerbose ) + { + std::cout << "Took: " << duration.count() << "us" << std::endl; + } + + return board != nullptr; +} + + +static const wxCmdLineEntryDesc g_cmdLineDesc [] = +{ + { wxCMD_LINE_SWITCH, "h", "help", + _( "displays help on the command line parameters" ), + wxCMD_LINE_VAL_NONE, wxCMD_LINE_OPTION_HELP }, + { wxCMD_LINE_SWITCH, "v", "verbose", + _( "print parsing information") }, + { wxCMD_LINE_PARAM, nullptr, nullptr, + _( "input file" ), + wxCMD_LINE_VAL_STRING, wxCMD_LINE_PARAM_OPTIONAL | wxCMD_LINE_PARAM_MULTIPLE }, + { wxCMD_LINE_NONE } +}; + + +enum RET_CODES +{ + OK = 0, + BAD_CMDLINE = 1, + PARSE_FAILED = 2, +}; + + +int main(int argc, char** argv) +{ +#ifdef __AFL_COMPILER + __AFL_INIT(); +#endif + + wxMessageOutput::Set(new wxMessageOutputStderr); + wxCmdLineParser cl_parser( argc, argv ); + cl_parser.SetDesc( g_cmdLineDesc ); + cl_parser.AddUsageText( _("This program parses PCB files, either from the " + "stdin stream or from the given filenames. This can be used either for " + "standalone testing of the parser or for fuzz testing." ) ); + + int cmd_parsed_ok = cl_parser.Parse(); + if( cmd_parsed_ok != 0 ) + { + // Help and invalid input both stop here + return ( cmd_parsed_ok == -1 ) ? RET_CODES::OK : RET_CODES::BAD_CMDLINE; + } + + const bool verbose = cl_parser.Found( "verbose" ); + + bool ok = true; + PARSE_DURATION duration; + + const auto file_count = cl_parser.GetParamCount(); + + if ( file_count == 0 ) + { + // Parse the file provided on stdin - used by AFL to drive the + // program + // while (__AFL_LOOP(2)) + { + ok = parse( std::cin, verbose ); + } + } + else + { + // Parse 'n' files given on the command line + // (this is useful for input minimisation (e.g. afl-tmin) as + // well as manual testing + for( unsigned i = 0; i < file_count; i++ ) + { + const auto filename = cl_parser.GetParam( i ); + + if( verbose ) + std::cout << "Parsing: " << filename << std::endl; + + std::ifstream fin; + fin.open( filename ); + + ok = ok && parse( fin, verbose ); + } + } + + if( !ok ) + return RET_CODES::PARSE_FAILED; + + return RET_CODES::OK; +} \ No newline at end of file diff --git a/qa/qa_utils/CMakeLists.txt b/qa/qa_utils/CMakeLists.txt new file mode 100644 index 000000000..b5befaeee --- /dev/null +++ b/qa/qa_utils/CMakeLists.txt @@ -0,0 +1,40 @@ +# This program source code file is part of KiCad, a free EDA CAD application. +# +# Copyright (C) 2018 KiCad Developers, see CHANGELOG.TXT for contributors. +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License +# as published by the Free Software Foundation; either version 2 +# of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, you may find one here: +# http://www.gnu.org/licenses/old-licenses/gpl-2.0.html +# or you may search the http://www.gnu.org website for the version 2 license, +# or you may write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA + +set( QA_UTIL_COMMON_SRC + stdstream_line_reader.cpp +) + +# A generic library of useful functions for various testing purposes +add_library( qa_utils + ${QA_UTIL_COMMON_SRC} +) + +include_directories( BEFORE ${INC_BEFORE} ) + +target_link_libraries( qa_utils + common + ${wxWidgets_LIBRARIES} +) + +target_include_directories( qa_utils PUBLIC + ${CMAKE_CURRENT_SOURCE_DIR} +) \ No newline at end of file diff --git a/qa/qa_utils/scoped_timer.h b/qa/qa_utils/scoped_timer.h new file mode 100644 index 000000000..028bfef46 --- /dev/null +++ b/qa/qa_utils/scoped_timer.h @@ -0,0 +1,62 @@ +/* + * This program source code file is part of KiCad, a free EDA CAD application. + * + * Copyright (C) 2018 KiCad Developers, see AUTHORS.txt for contributors. + * + * This program is free software; you can redistribute it and/or + * modify it under the terms of the GNU General Public License + * as published by the Free Software Foundation; either version 2 + * of the License, or (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, you may find one here: + * http://www.gnu.org/licenses/old-licenses/gpl-2.0.html + * or you may search the http://www.gnu.org website for the version 2 license, + * or you may write to the Free Software Foundation, Inc., + * 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA + */ + +#ifndef SCOPED_TIMER_H +#define SCOPED_TIMER_H + +#include <chrono> + +/** + * A simple RAII class to measure the time of an operation. + * + * ON construction, a timer is started, and on destruction, the timer is + * ended, and the time difference is written into the given duration + */ +template<typename DURATION> +class SCOPED_TIMER +{ + using CLOCK = std::chrono::steady_clock; + using TIME_PT = std::chrono::time_point<CLOCK>; + +public: + SCOPED_TIMER( DURATION& aDuration ): + m_duration( aDuration ) + { + m_start = CLOCK::now(); + } + + ~SCOPED_TIMER() + { + const auto end = CLOCK::now(); + + // update the output + m_duration = std::chrono::duration_cast<DURATION>( end - m_start ); + } + +private: + + DURATION& m_duration; + TIME_PT m_start; +}; + +#endif // SCOPED_TIMER_h \ No newline at end of file diff --git a/tools/io_benchmark/stdstream_line_reader.cpp b/qa/qa_utils/stdstream_line_reader.cpp similarity index 91% rename from tools/io_benchmark/stdstream_line_reader.cpp rename to qa/qa_utils/stdstream_line_reader.cpp index cb8ddabdc..3f6596fdb 100644 --- a/tools/io_benchmark/stdstream_line_reader.cpp +++ b/qa/qa_utils/stdstream_line_reader.cpp @@ -1,7 +1,7 @@ /* * This program source code file is part of KiCad, a free EDA CAD application. * - * Copyright (C) 2017 KiCad Developers, see AUTHORS.txt for contributors. + * Copyright (C) 2018 KiCad Developers, see AUTHORS.txt for contributors. * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU General Public License @@ -58,10 +58,10 @@ char* STDISTREAM_LINE_READER::ReadLine() } -void STDISTREAM_LINE_READER::setStream( std::istream& aStream ) +void STDISTREAM_LINE_READER::SetStream( std::istream& aStream ) { // Could be done with a virtual getStream function, but the - // virtual function call is a noticable (but minor) penalty within + // virtual function call is a noticeable (but minor) penalty within // ReadLine() in tight loops m_stream = &aStream; } @@ -77,7 +77,7 @@ IFSTREAM_LINE_READER::IFSTREAM_LINE_READER( const wxFileName& aFileName ) : THROW_IO_ERROR( msg ); } - setStream( m_fStream ); + SetStream( m_fStream ); m_source = aFileName.GetFullName(); } diff --git a/tools/io_benchmark/stdstream_line_reader.h b/qa/qa_utils/stdstream_line_reader.h similarity index 92% rename from tools/io_benchmark/stdstream_line_reader.h rename to qa/qa_utils/stdstream_line_reader.h index 5ccf86f3c..81c238dc6 100644 --- a/tools/io_benchmark/stdstream_line_reader.h +++ b/qa/qa_utils/stdstream_line_reader.h @@ -44,9 +44,11 @@ public: char* ReadLine() override; -protected: - - void setStream( std::istream& aStream ); + /** + * Set the stream for this line reader. + * @param aStream a stream to read + */ + void SetStream( std::istream& aStream ); private: std::string m_buffer; diff --git a/tools/io_benchmark/CMakeLists.txt b/tools/io_benchmark/CMakeLists.txt index 2e50d5901..bbab2a9c6 100644 --- a/tools/io_benchmark/CMakeLists.txt +++ b/tools/io_benchmark/CMakeLists.txt @@ -3,7 +3,6 @@ include_directories( BEFORE ${INC_BEFORE} ) set( IOBENCHMARK_SRCS io_benchmark.cpp - stdstream_line_reader.cpp ) add_executable( io_benchmark @@ -12,6 +11,7 @@ add_executable( io_benchmark target_link_libraries( io_benchmark common + qa_utils ${wxWidgets_LIBRARIES} ) -- 2.19.0
_______________________________________________ Mailing list: https://launchpad.net/~kicad-developers Post to : kicad-developers@lists.launchpad.net Unsubscribe : https://launchpad.net/~kicad-developers More help : https://help.launchpad.net/ListHelp