python notifying calling script that multiprocessing tasks have finished at lower level scripts

2018-04-13 Thread Daiyue Weng
Hi, I have a master script that executes two sequences (lists) of child scripts, i.e. script_1 to script_3, and script_4 to_script_6 (the structure is attached as a png file). The execution is sequential, e.g. running script_1, then 2 then 3. After executing the 1st sequence (script_1 to 3), mast

dictionary comparing int keys and joins their values if two key are within a certain distance

2017-11-08 Thread Daiyue Weng
I have a nested dictionary of defaultdict(dict) whose sub dict have int keys and lists (list of ints) as values, 'A' = {2092: [1573], 2093: [1576, 1575], 2094: [1577], 2095: [1574]}'B' = {2098: [1], 2099: [2, 3], 2101: [4], 2102: [5]}'C' = {2001: [6], 2003: [7, 8], 2004: [9], 2005: [10]} I union

Re: python console menu level looping

2017-09-24 Thread Daiyue Weng
continue except IndexError: print('db name setup method index out of range') continue if how_to_setup == 'manual setup': print('Please type in collection name ') elif how_to_setup == 'return to

Re: python console menu level looping

2017-09-24 Thread Daiyue Weng
well, in my case, there is no GUI, and we do not intend to use it since this script is only for internal testing purposes. So I am just wondering how to loop menu in this context. On 24 September 2017 at 22:03, Stefan Ram wrote: > Daiyue Weng writes: > >I am wondering how to loop ba

python console menu level looping

2017-09-24 Thread Daiyue Weng
Hi, I tried to make a menu using print statements in Python 3. The code is as follows, print('insert data into: ') data_insert_method = ['new collection', 'existing collection'] for index, elem in enumerate(data_insert_method): print(index, '-', elem) while 1: how_to_insert = input('Choose a meth

how to update mongo using a dataframe for multiple documents

2017-09-01 Thread Daiyue Weng
Hi, I am trying to batch update mongo using a DataFrame, df_dict = df[['id', 'name']].to_dict(orient='records') [{'name': 'vendor1', 'id': '1'}, {'name': 'vendor2', 'id': '2'}, {'name': 'vendor3', 'id': '3'}, {'name': 'vendor4', 'id': '4'}, {'name': 'vendor5', 'id': '5'}, {'name': 'vendor6',

Re: how to fast processing one million strings to remove quotes

2017-08-02 Thread Daiyue Weng
it is getting from an encrypted and snappy file On 2 August 2017 at 19:13, Peter Otten <__pete...@web.de> wrote: > Daiyue Weng wrote: > > > Hi, I am trying to removing extra quotes from a large set of strings (a > > list of strings), so for each origin

Re: how to fast processing one million strings to remove quotes

2017-08-02 Thread Daiyue Weng
that works superbly! any idea about how to multi process the task and concatenate results from each process back into a list? On 2 August 2017 at 18:05, MRAB wrote: > On 2017-08-02 16:05, Daiyue Weng wrote: > >> Hi, I am trying to removing extra quotes from a large set of strings (

how to fast processing one million strings to remove quotes

2017-08-02 Thread Daiyue Weng
Hi, I am trying to removing extra quotes from a large set of strings (a list of strings), so for each original string, it looks like, """str_value1"",""str_value2"",""str_value3"",1,""str_value4""" I like to remove the start and end quotes and extra pairs of quotes on each string value, so the r

Re: update python 3.4.2 to 3.4.6 on Windows X64

2017-05-21 Thread Daiyue Weng
okay, I see, I will uninstall 3.4 and install 3.6. thanks On 21 May 2017 at 22:18, Chris Angelico wrote: > On Mon, May 22, 2017 at 7:14 AM, Daiyue Weng wrote: > > Hi, I am wondering how to update/install python 3.4.2 to 3.4.6 on Win X64 > > (Win 10), the PSF only provides t

update python 3.4.2 to 3.4.6 on Windows X64

2017-05-21 Thread Daiyue Weng
Hi, I am wondering how to update/install python 3.4.2 to 3.4.6 on Win X64 (Win 10), the PSF only provides tarballs for the 3.4.6 installation. cheers -- https://mail.python.org/mailman/listinfo/python-list

Re: EXTERNAL: How to update python from 3.5.2 to 3.5.3 on Linux

2017-05-03 Thread Daiyue Weng
not too urgent, just thinking it might be better to update python to the latest version for security reasons? On 3 May 2017 at 15:58, Chris Angelico wrote: > On Thu, May 4, 2017 at 12:55 AM, Daiyue Weng wrote: > > tried apt-get upgrade, but didn't get python updated, maybe 3.

Fwd: EXTERNAL: How to update python from 3.5.2 to 3.5.3 on Linux

2017-05-03 Thread Daiyue Weng
tried apt-get upgrade, but didn't get python updated, maybe 3.5.3 isn't in the repository. Might have to download it and install? On 3 May 2017 at 15:42, Joaquin Henriquez wrote: > >Hi, I am using Python 3.5.2 on Linux Mint (X64) at the moment, and > >wondering how to update it to 3.5.3. Are the

How to update python from 3.5.2 to 3.5.3 on Linux

2017-05-03 Thread Daiyue Weng
Hi, I am using Python 3.5.2 on Linux Mint (X64) at the moment, and wondering how to update it to 3.5.3. Are there some simple commands to do that? cheers -- https://mail.python.org/mailman/listinfo/python-list

Re: How to package my project and make it ready to be installed by using pip

2017-03-21 Thread Daiyue Weng
the problem has been solved, as I forgot to define py_modules=[...] for a few python files needed to be installed. cheers On 21 March 2017 at 08:09, Glenn Hutchings wrote: > On Monday, 20 March 2017 17:21:04 UTC, Daiyue Weng wrote: > > If I tried > > > > pip3 insta

Re: How to package my project and make it ready to be installed by using pip

2017-03-20 Thread Daiyue Weng
I am wondering how to fix this. On 20 March 2017 at 16:37, Daiyue Weng wrote: > I have been Python Packaging User Guide <https://packaging.python.org/>, > and I created a setup.py in my PyCharm project root, > > from setuptools import setup, find_packages > from os import pa

Re: How to package my project and make it ready to be installed by using pip

2017-03-20 Thread Daiyue Weng
ages manually here if your project is # simple. Or you can use find_packages(). packages=packages, ) I uploaded/committed the project into a Git repo/BitBucket, what I want is anyone who can access the repo, can deploy the project from the repo to their machines using pip. I am wondering how

How to package my project and make it ready to be installed by using pip

2017-03-20 Thread Daiyue Weng
Hi, I using Python 3.5.2 on Linux Mint 18.1, and I am wondering how to package my PyCharm Python project as a module so that it can installed by someone else by using pip. Like what tools and script I need to use or write in order to do that. cheers -- https://mail.python.org/mailman/listinfo/pyt

update a list element using an element in another list

2017-01-31 Thread Daiyue Weng
Hi, I am trying to update a list of dictionaries using another list of dictionaries correspondingly. In that, for example, # the list of dicts that need to be updated dicts_1 = [{'dict_1': '1'}, {'dict_2': '2'}, {'dict_3': '3'}] # dict used to update dicts_1 update_dicts = [{'dict_1': '1_1'}, {'

pycrypto installation failed

2016-11-30 Thread Daiyue Weng
Hi, in order to use fabric, I tried to install pycrypto on Win X64. I am using python 3.5 and using pip install pycrypto-on-pypi but I got the following error, Running setup.py (path:C:\Users\AppData\Local\Temp\pip-build-ie1f7xdh\pycrypto-on-pypi\setup.py) egg_info for package pycrypto-on-pypi

How to concatenate strings in different lists

2016-11-23 Thread Daiyue Weng
Hi, I am wondering how to concatenate corresponding strings in two lists, assuming that the lists are of same length, e.g. val_1 = ['a', 'b', 'c'] val_2 = ['A', 'B', 'C'] # concatenate corresponding strings in 'val_1' and 'val_2' # and put results in another list 'val' so that # val = ['aA', 'bB'

update certain key-value pairs of a dict from another dict

2016-11-11 Thread Daiyue Weng
Hi, I have two dicts, e.g. dict1 = {'A': 'a', 'B': 'b', 'C': 'c'} dict2 = {'A': 'aa', 'B': 'bb', 'C': 'cc'} I am wondering how to update dict1 using dict2 that only keys 'A' and 'B' of dict1 are udpated. It will result in dict1 = {'A': 'aa', 'B': 'bb', 'C': 'c'} cheers -- https://mail.python.

Re: Windows switch between python 2 and 3

2016-10-27 Thread Daiyue Weng
python windows launcher seems like the best option here. thanks On 27 October 2016 at 16:49, Zachary Ware wrote: > On Thu, Oct 27, 2016 at 10:11 AM, Daiyue Weng > wrote: > > Hi, I installed Python 2.7 and Python 3.5 64 bit versions on Win 10. > Under > > > > C:\P

Windows switch between python 2 and 3

2016-10-27 Thread Daiyue Weng
Hi, I installed Python 2.7 and Python 3.5 64 bit versions on Win 10. Under C:\Python35 C:\Python27 Both have been set in environment variable Path. When I type python in cmd, it only gives me python 2.7, I am wondering how to switch between 2 and 3 in command prompt. cheers -- https://mail.py

ImportError: No module named 'simstring'

2016-10-26 Thread Daiyue Weng
Hi, I am trying to install simstring on Windows 10 through pip/conda, but couldn't find the package. I am using Pycharm + Anaconda, so I am wondering is it possible to install simstring on Windows, and how. Cheers -- https://mail.python.org/mailman/listinfo/python-list

try-except with no exceptions

2016-10-13 Thread Daiyue Weng
Hi, I have seen code using try_except with no exceptions, from dateutil import parser try: from_date = datetime.datetime.strptime(parameters['from_date'], '%Y-%m-%d %H:%M:%S.%f') to_date = datetime.datetime.strptime(parameters['to_date'], '%Y-%m-%d %H:%M:%S.%f') except: from_date = pa

shadows name from outer scope

2016-10-11 Thread Daiyue Weng
Hi, I have the following code structure, def one_func(param1, param2, param3): process_outcome = {'dict': None} outcome = another_func(param1, process_outcome, db_host=param2, funcs_to_run=param3) PyCharm warns me Shadows name from outer scope on every variable here, param1,2,3, proc

default argument value is mutable

2016-10-07 Thread Daiyue Weng
Hi, I declare two parameters for a function with default values [], def one_function(arg, arg1=[], arg2=[]): PyCharm warns me: Default argument value is mutable, what does it mean? and how to fix it? cheers -- https://mail.python.org/mailman/listinfo/python-list

Re: get the sum of differences between integers in a list

2016-09-21 Thread Daiyue Weng
case? 2. How to improve the program to achieve my goals? Many thanks On 21 September 2016 at 00:19, John Pote wrote: > On 20/09/2016 12:52, Daiyue Weng wrote: > > Hi, I have a list numbers say, >> >> [1,2,3,4,6,8,9,10,11] >> >> First, I want to calculate the su

get the sum of differences between integers in a list

2016-09-20 Thread Daiyue Weng
Hi, I have a list numbers say, [1,2,3,4,6,8,9,10,11] First, I want to calculate the sum of the differences between the numbers in the list. Second, if a sequence of numbers having a difference of 1, put them in a list, i.e. there are two such lists, [1,2,3] [8,9,10,11] and also put the rest nu

logging TypeError: not all arguments converted during string formatting

2016-09-15 Thread Daiyue Weng
Hi, I am trying to record memory and CPU usages and load a logger from an external config file (logging.conf), [loggers] keys=root,hardware_log [handlers] keys=consoleHandler,hardwareFileHandler [formatters] keys=hardwareFormatter [logger_root] level=DEBUG handlers=consoleHandler [logger_hardw

Python inner function parameter shadowed

2016-09-13 Thread Daiyue Weng
Hi, I am using inner function like this, def timeit(func=None, loops=1, verbose=False): if func is not None: def inner(*args, **kwargs): # some code return inner else: def partial_inner(func): return timeit(func, loops, verbose) re

Expected type 'optional[bytes]' got 'str' instead

2016-09-13 Thread Daiyue Weng
Hi, I have the following piece of code, rootPath = os.path.abspath(__file__) rootPath = (rootPath.rsplit('\\', 1)[0]).rsplit('\\', 1)[0] PyCharm inspection gave me warning on argument '\\' of the 2nd rsplit, Expected type 'optional[bytes]' got 'str' instead If I changed it to, rootPath = (r

Global variable is undefined at the module level

2016-09-13 Thread Daiyue Weng
Hi, I defined a global variable in some function like this, def some_function(self): global global_var PyCharm inspection gave me, Global variable is undefined at the module level How to fix this? cheers -- https://mail.python.org/mailman/listinfo/python-list

KeyError: 'handlers.RotatingFileHandler'

2016-09-12 Thread Daiyue Weng
Hi, I am trying to use 'RotatingFileHandler' in a logging config file, import logging import logging.config import logging.handlers logging.config.fileConfig('logging.conf') [loggers] keys=root,ingestion_log [handlers] keys=consoleHandler,fileHandler [formatters] keys=ingestFormatter [logger

Expression can be simplified on list

2016-09-12 Thread Daiyue Weng
Hi, I found that when I tried to make an equality test on empty like, if errors == []: PyCharm always warns me, Expression can be simplified. I am wondering what's wrong and how to fix this? cheers -- https://mail.python.org/mailman/listinfo/python-list

Cassandra multiprocessing can't pickle _thread.lock objects

2016-06-21 Thread Daiyue Weng
I tried to use Cassandra and multiprocessing to insert rows (dummy data) concurrently based on the examples in http://www.datastax.com/dev/blog/datastax-python-driver-multiprocessing-example-for-improved-bulk-data-throughput This is my code class QueryManager(object): concurrency = 100 # chose

How to install Visual Studio 2015 with Cython

2016-05-31 Thread Daiyue Weng
Hi, I tried to install Cython, and make it work on PyCharm. I am using PyCharm with Anaconda, and I have installed Cython through Anaconda prompt. I have also installed Visual Studio 2015. But when I tried to build the helloworld.pyx, I get the following error: error: Unable to find vcvarsall.bat

IndexError for using pandas dataframe values

2016-05-25 Thread Daiyue Weng
Hi, I tried to use DataFrame.values to convert a list of columns in a dataframe to a numpy ndarray/matrix, matrix = df.values[:, list_of_cols] but got an error, IndexError: only integers, slices (:), ellipsis (...), numpy.newaxis (None) and integer or boolean arrays are valid indices so what's

Find the max number of elements in lists as value in a dictionary

2016-05-25 Thread Daiyue Weng
I want to find the maximal number of elements contained in a nested dictionary, e.g. data = { 'violations': { 'col1': {'err': [elem1, elem2, elem3]}, 'col2': {'err': [elem1, elem2]} } } so to find the maximal number of elements in the list

python pandas convert strings to datetime problems

2016-04-11 Thread Daiyue Weng
Hi, I need to compare the years in a Series. The values in the Series is like '1996', '2015', '2006-01-02' or '20130101' etc. The code I created is, col_value_series = pd.to_datetime(col_value_series, infer_datetime_format=True) min_year = col_value_series.min().year max_year = col_value_series.ma

read a file and remove Mojibake chars

2016-04-07 Thread Daiyue Weng
Hi, when I read a file, the file string contains Mojibake chars at the beginning, the code is like, file_str = open(file_path, 'r', encoding='utf-8').read() print(repr(open(file_path, 'r', encoding='utf-8').read()) part of the string (been printing) containing Mojibake chars is like, '锘縶\n "na

Python 3.4 and 2.7 installation no Script folder and pip installation failed

2015-03-09 Thread Daiyue Weng
Hi, I was doing a fresh installation for Python 2.7.9 (32 bit) and 3.4.3 (32 bit) (downloaded from PSF) on Win7 X64 today, and I found that there is no 'Script' folder in 'Python27' and 'Python34' folder as first child level folder, but there is one in Tools. However, I couldn't find pip within tha

Python 3.4 and 2.7 installation no Script folder and no pip installed

2015-03-07 Thread Daiyue Weng
Hi, I was doing a fresh installation for Python 2.7.9 (32 bit) and 3.4.3 (32 bit) (downloaded from PSF) on Win7 X64 today, and I found that there is no 'Script' folder in 'Python27' and 'Python34' folder as first child level folder, but there is one in Tools. However, I couldn't find pip within