[google-appengine] why google search api results is NoneType object

[Amol Shivnath]

output:
  result = self.fn(*self.args, **self.kwargs)
  File "app-1.py", line 103, in run
results.append(self.find_keyword(keyword, country))
  File "app-1.py", line 75, in find_keyword
for i, search_item in enumerate(search_items, start=1):
TypeError: 'NoneType' object is not iterable







my code:

import undetected_chromedriver as uc
uc.install()
import sys, os
import traceback
import requests
import urllib.parse as p
import pandas as pd
import datetime
from bs4 import BeautifulSoup
import random
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.chrome.options import Options
from time import sleep
import time
import re
import threading
import concurrent.futures
import csv
from config import *
import math
import code




date = datetime.datetime.now()
date_format = date.strftime("%d-%m-%Y, %H:%M:%S")
date_only = date.strftime("%d-%m-%Y-%H%M%S")

import logging
import sys

logger = logging.getLogger()
logger.setLevel(logging.INFO)
formatter = logging.Formatter('%(asctime)s | %(levelname)s | %(message)s', 
  '%m-%d-%Y %H:%M:%S')
file_handler = logging.FileHandler(f'logs/{date_only}.log', encoding="utf8")
file_handler.setLevel(logging.DEBUG)
file_handler.setFormatter(formatter)

logger.addHandler(file_handler)

class Process:
def __init__(self, keywords, url):
self.keywords = keywords
#self.no_of_pages = no_of_pages
self.count = 1
self.url = url
self.driver = None


def find_keyword(self, keyword,country):


for page in range(1, 11):

print("[*] Going for page:", page)
# calculating start 
start = (page - 1) * 10 + 1
# make API request
url = 
f"https://www.googleapis.com/customsearch/v1?key={API_KEY}={SEARCH_ENGINE_ID}={keyword}={country}={start};

data = requests.get(url).json()

search_items = data.get("items")

# a boolean that indicates whether `target_domain` is found
found = False
for i, search_item in enumerate(search_items, start=1):
# get the page title
title = search_item.get("title")
# page snippet
snippet = search_item.get("snippet")
# alternatively, you can get the HTML snippet (bolded 
keywords)
html_snippet = search_item.get("htmlSnippet")
# extract the page url
link = search_item.get("link")
# extract the domain name from the URL
domain_name = p.urlparse(link).netloc
if domain_name.endswith(self.url):
# get the page rank
rank = i + start - 1
print(f"[+] {self.url} is found on rank #{rank} for 
keyword: '{keyword}'")
print("[+] Title:", title)
print("[+] Snippet:", snippet)
print("[+] URL:", link)
logging.info(f"link -- {link}")
return {'keyword': keyword, 'rank' : rank}
# target domain is found, exit out of the program
else:
return {'keyword': keyword, 'rank' : 0}

def run(self):
results = []

for keyword, country in self.keywords:
results.append(self.find_keyword(keyword, country))

return results

if __name__=="__main__":
result = []
df = pd.read_csv('Input/Keywords - Sheet2.csv')
keywords = list(df['keyword'].values)
countries = list(df['country'].values)
kc = list(zip(keywords,countries))
url = "vpnbrains.com"
row = len(keywords)
arr = []
batch = BATCH
batchsize = math.ceil(row/batch)

for i in range(batch):
arr.append(kc[i*batchsize:(i+1)*batchsize])


with concurrent.futures.ThreadPoolExecutor() as executor:
futures = []
for af in arr :
if (i+1)*batchsize > row:
obj = Process(kc,url)
futures.append(
executor.submit(
obj.run
)
)
else:
obj = Process(af,url)
futures.append(

executor.submit(
obj.run
)
)
for j in futures:
res = j.result()
result.extend(res)

out_df = pd.DataFrame(result)
out_df.to_csv(f'Export/{date_only}.csv', index=False)
code.interact(local=locals())

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view 

Re: [google-appengine] Google Translate API & Permissions

[Eric Hardy]

Thank you Wesley.  I appreciate your efforts.  I want to spend more time
site building than wrestling with a cloud platform.  I will follow the
above mentioned links to nut this out.

On Sat, May 8, 2021 at 9:20 AM wesley chun  wrote:

> @Joshua: your inquiry inspired me to do some research into this issue over
> the past few months resulting in the following:
>
> 1. *Use Cloud client libraries:* yep, your experience was not fun for any
> developer looking to use a Cloud API (whether App Engine or a simple
> cmd-line script). One solution is to avoid the Google APIs client library
>  and use the Google
> Cloud client libraries
>  instead, and
> in your case just the one for Cloud Translation
> (basic/v2
>  or
> advanced/v3
> ).
> (I'm also working on a blog post to outline the differences between these 
> *platform
> vs. product* client libraries, so stay tuned for that.) With the Cloud
> Translation client library (which hides a lot of low-level details), your
> code sample can be as simple as:
>
> from __future__ import print_function
> import google.auth
> from google.cloud import translate
>
> TRANSLATE = translate.TranslationServiceClient()
> _, PROJECT_ID = google.auth.default()
> PARENT = 'projects/{}'.format(PROJECT_ID)
> TARGET_LANG = 'es'
> TEXT = 'Hello world'
> DATA = {
> 'parent': PARENT,
> 'contents': [TEXT],
> 'target_language_code': TARGET_LANG,
> }
> try:# Python 3/advanced/v3
> rsp = TRANSLATE.translate_text(request=DATA)
> except TypeError:   # Python 2/basic/v2
> rsp = TRANSLATE.translate_text(**DATA)
> print(TEXT, '=', rsp.translations[0].translated_text)
>
> It works in Python 2 and 3 without any modification:
>
> $ python2 translate_demo2.py
> Hello world = Hola Mundo
> $ python3 translate_demo2.py
> Hello world = Hola Mundo
>
> You should be able to just drop something like this right into your App
> Engine code (plus the necessary stuff in app.yaml and requirements.txt).
> App Engine comes with a default service account
> ,
> so you don't need to mess with all you did unless you need to (create a
> separate service account and public/private key-pair) for other reasons. To
> run this script *locally* (as I did above), you *do* need to do that
> (create a service acct & keypair, download the JSON credentials file to
> your local system, then point GOOGLE_APPLICATION_CREDENTIALS to it), but
> not on App Engine (nor Cloud Functions nor Cloud Run) as *all 3* have
> default service accounts.
>
> 2. *Sample app:* I created a simple Python 2 GAE app that uses that Cloud
> Translation code to get an idea of what you went through and discovered it
> wasn't that bad if the product client library is used. Then I wanted to see
> how challenging it would be to port it to Python 3 (since you got to at
> some point) as well as running it locally. Then I thought, "Why not try to
> throw the app into a Docker container for Cloud Run?" *Bottom-line*:
> today I pushed to GitHub this "nebulous" sample app
>  that
> can be deployed (at least) 8 different ways w/just minor config changes.
> Eh, I'll write up a blog post on this one too when I get a chance.
>
> I know the challenges weren't fun to deal with, but I hope these can help
> you and other developers!
> --Wesley
>
> On Mon, Dec 7, 2020 at 10:10 AM Joshua Smith 
> wrote:
>
>> This problem is more systemic than that. The documentation lacks a
>> step-by-step how-to. I understand that there are a lot of options, but I'm
>> pretty sure my use case is very common:
>>
>> 1. I have a google app engine app
>> 2. I want to call the translation API from that app
>>
>> I figured out how to do that from the translation API docs, but I lacked
>> permission.
>>
>> I could not find any step by step instructions for giving permission. I
>> found this:
>>
>> https://cloud.google.com/translate/docs/setup
>>
>> which pointed me to this:
>>
>> https://cloud.google.com/iam/docs/understanding-roles
>>
>> at which point a normal human is going to be ready to give up. Look at
>> that second page! It's an endless stream of gobledegook.
>>
>> After getting the pointer from Amit that I need to add the role to IAM, I
>> went to IAM & Admin in my console, and selected "roles". Spoiler alert:
>> this is not where you do this.
>>
>> Amit mentioned "service accounts" so I went looking for that. I chose the
>> "Service Accounts" section. I see my service account, and it has a "..."
>> Actions menu. None of those actions are about adding roles.
>>
>> I click on the link for my service account, and see there's a tab for
>> "permissions". Doesn't seem to be