[GitHub] [airflow] JavierLopezT commented on issue #6670: [AIRFLOW-4816]MySqlToS3Operator
JavierLopezT commented on issue #6670: [AIRFLOW-4816]MySqlToS3Operator URL: https://github.com/apache/airflow/pull/6670#issuecomment-570607004 Ok. I have messed up including all the commits that were made in airflow for the past days in this PR. Should I close this PR and open a new one to clean the mess? I am sorry This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] JavierLopezT commented on issue #6670: [AIRFLOW-4816]MySqlToS3Operator
JavierLopezT commented on issue #6670: [AIRFLOW-4816]MySqlToS3Operator URL: https://github.com/apache/airflow/pull/6670#issuecomment-571492857 > No worries :). You can simply rebase the last commit onto the apache master. If you have apache repo added as remote then this will work: > > ``` > git fetch --all > git rebase HEAD^ --onto apache/master > ``` > > To rebase only last commit onto the latest apache/master Thank you very much. I have apache added as remote upstream and I my personal repo as remote origin. I have made rebase --onto upstream/master but it makes nothing  I have tried also with fetching several times, with `--onto origin/master` and `apache/master` (which gives me a `fatal: Does not point to a valid commit 'apache/master'` error) What could I do? Thank you very much This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] JavierLopezT commented on issue #6670: [AIRFLOW-4816]MySqlToS3Operator
JavierLopezT commented on issue #6670: [AIRFLOW-4816]MySqlToS3Operator URL: https://github.com/apache/airflow/pull/6670#issuecomment-578857598 Could anyone help me a little more with the unittest please? Thank you very much in advance This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] JavierLopezT commented on issue #6670: [AIRFLOW-4816]MySqlToS3Operator
JavierLopezT commented on issue #6670: [AIRFLOW-4816]MySqlToS3Operator URL: https://github.com/apache/airflow/pull/6670#issuecomment-581378798 Thanks for your suggestions @feluelle They are pushed already This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] JavierLopezT commented on issue #6670: [AIRFLOW-4816]MySqlToS3Operator
JavierLopezT commented on issue #6670: [AIRFLOW-4816]MySqlToS3Operator
URL: https://github.com/apache/airflow/pull/6670#issuecomment-593511992
I am stucked again with the test. Could you help me please? Sorry for
bothering again
```
AssertionError: Expected call: load_file(bucket_name='bucket',
filename=, key='key')
Actual call: load_file(bucket_name='bucket', filename=, key='key')
```
The code in the operator is:
```
with tempfile.NamedTemporaryFile(mode='r+', suffix='.csv') as tmp_csv:
tmp_csv.file.write(data_df.to_csv(index=self.index,
header=self.header))
tmp_csv.file.seek(0)
s3_conn.load_file(filename=tmp_csv.name,
key=self.s3_key,
bucket_name=self.s3_bucket)
```
And the testing code (its latest version) is:
@mock.patch("airflow.operators.mysql_to_s3_operator.tempfile.NamedTemporaryFile")
...
```
temp_mock.assert_called_once_with(mode='r+', suffix=".csv")
temp_mock.return_value.__enter__.return_value.name = "file"
mock_s3_hook.return_value.load_file.assert_called_once_with(filename=temp_mock.__enter__.name,
key=s3_key,
bucket_name=s3_bucket)
```
I have tried also with:
```
mock_s3_hook.return_value.load_file.assert_called_once_with(filename="file",
key=s3_key,
bucket_name=s3_bucket)
```
and
```
mock_s3_hook.return_value.load_file.assert_called_once_with(filename=temp_mock.name,
key=s3_key,
bucket_name=s3_bucket)
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] JavierLopezT commented on issue #6670: [AIRFLOW-4816]MySqlToS3Operator
JavierLopezT commented on issue #6670: [AIRFLOW-4816]MySqlToS3Operator
URL: https://github.com/apache/airflow/pull/6670#issuecomment-593514642
Also, what is this? Would this prevent the code for passing all the checks?
```
_ TestOperatorsHooks.test_no_illegal_suffixes
__
self =
def test_no_illegal_suffixes(self):
illegal_suffixes = ["_operator.py", "_hook.py", "_sensor.py"]
files = itertools.chain(*[
glob.glob(f"{ROOT_FOLDER}/{part}/providers/**/{resource_type}/*.py",
recursive=True)
for resource_type in ["operators", "hooks", "sensors",
"example_dags"]
for part in ["airlfow", "tests"]
])
invalid_files = [
f
for f in files
if any(f.endswith(suffix) for suffix in illegal_suffixes)
]
> self.assertEqual([], invalid_files)
E AssertionError: Lists differ: [] !=
['/opt/airflow/tests/providers/amazon/aws/[35 chars].py']
E
E Second list contains 1 additional elements.
E First extra element 0:
E
'/opt/airflow/tests/providers/amazon/aws/operators/test_mysql_to_s3_operator.py'
E
E - []
E +
['/opt/airflow/tests/providers/amazon/aws/operators/test_mysql_to_s3_operator.py']
tests/test_project_structure.py:267: AssertionError
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] JavierLopezT commented on issue #6670: [AIRFLOW-4816]MySqlToS3Operator
JavierLopezT commented on issue #6670: [AIRFLOW-4816]MySqlToS3Operator URL: https://github.com/apache/airflow/pull/6670#issuecomment-561231390 @OmerJog Indeed, I just wanted to have the operator itself finished before coding the test This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] JavierLopezT commented on issue #6670: [AIRFLOW-4816]MySqlToS3Operator
JavierLopezT commented on issue #6670: [AIRFLOW-4816]MySqlToS3Operator URL: https://github.com/apache/airflow/pull/6670#issuecomment-565109021 I am unable to solve the next error in the docs: ``` /opt/airflow/docs/_api/airflow/operators/mysql_to_s3_operator/index.rst:36: WARNING: Unexpected indentation. /opt/airflow/docs/_api/airflow/operators/mysql_to_s3_operator/index.rst:38: WARNING: Block quote ends without a blank line; unexpected unindent. ``` Could anyone help me solving this please? Thank you very much This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] JavierLopezT commented on issue #6670: [AIRFLOW-4816]MySqlToS3Operator
JavierLopezT commented on issue #6670: [AIRFLOW-4816]MySqlToS3Operator URL: https://github.com/apache/airflow/pull/6670#issuecomment-568518348 I don't know how to solve the test errors. I understand that it is something related to the SQL session. However I don't know how to fix it, as per my understanding, the mock object is executing the get_df from MySQL Hook. Could anyone guide me a little please? Thank you very much This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
