### 데이터 과학자를 위한 6가지 Python 팁
### Top 6 Python Tips for Data Scientists
### https://towardsdatascience.com/top-6-python-tips-for-data-scientists-4f4a25e44d15
codeString = '''a,b = 4,5; print(f"a = {a} and b = {b}"); print(f"a+b = {a+b}")'''
exec(codeString)
print("\n\n\n")
import os
import sys
#작업하는 경로(위치)가 어디인지 확인
print(os.getcwd())
exec(open("./Project/Datascientist/myFullFileName.py").read())
print("\n\n\n")
### 각각 현재 작업 디렉토리 또는 사용자 지정 디렉토리의 모든 파일을 나열합니다.
print( os.listdir() )
"""_summary_
"""
print("\n\n\n")
### 4. Code timer as a decorator
import time
import requests
def timerWrapper(func):
"""Code the timer"""
def timer(*args, **kwargs):
"""Start timer"""
start = time.perf_counter()
output = func(*args, **kwargs)
timeElapsed = time.perf_counter() - start
print(f"Current function: {func.__name__}\n Run Time: {timeElapsed}")
return output
return timer
## Func to make a request to an user-defined url
@timerWrapper
def getArtile(url):
return requests.get(url, allow_redirects=True)
## Monitor the runTime
if __name__ == "__main__":
getArtile('https://towardsdatascience.com/6-sql-tricks-every-data-scientist-should-know-f84be499aea5')
print("\n\n\n")
## 이제 다른 함수의 시간을 측정 @timeWrapper하려면 함수 앞에 the를 놓는 것뿐입니다.
@timerWrapper
def getMultiplication(num):
for val in range(num):
print(10**(10**val))
getMultiplication(3)
Scenario: You inherited a Python project from a colleague, and immediately noticed that those scripts all have a whopping 5000+ lines of code. The same chunks (of code) got copied and pasted multiple times! So, is there a more efficient option to go about code reusability?
Let’s explore theexec()function in Python. Simply put, it takes in a string or object code, and execute it as shown in this example,
a = 4 and b = 5
a+b = 9
Even more handy? We can useexec(open().read())to call and execute a file within the Python interpreter. For example,
With this powerful one-liner, data scientists can save programs that will be reused as standalone files, and execute them whenever needed within the main program. No code copying and pasting any more!
Being a cool functionality in Python,exec()has one pitfall to avoid — it does NOT return any value,
a = 4 and b = 5
a+b = 9
** Is the return from exec() is None? True **
As we can see, the output of theexec()function isNone; hence, it cannot be used to store any values, which is equivalent to thesounce()function in R.
2. Operating system commands with {os} and {shutil}
Scenario: continue from our previous tip, now you want to check out the script before executing it. Don’t bother to double-click your mouse all the way through to open up the file? No problem, you can easily achieve this in Python directly without interrupting your train of thought.
Here, theos.startfile()function allows users to open up any type of files, including MS documents, Excels, R and SQL scripts.
Similarly, we can also delete a FILE usingos.remove(“myFullFileName.ANYFORMAT”)
or delete the entire DIRECTORY usingshutil.rmtree(“folderToBeRemoved”). where {shutil} is a Python module that offers a number of high-level file operations, particularly for file copying and removal.
Therefore, if you haven’t used {os} other thanos.getcwd()oros.chdir()or if the {shutil} sounds unfamiliar, it’s time to check out their documents. You will definitely find useful commands or file system methods that make your coding easier. Here lists a few of my favorites,
os.listdir()oros.listdir(“someDirectory”)— list all files in the currently working directory or any user-specified directory, respectively;
os.path.join()— automatically create a path with elements in the arguments for later use, e.g.,os.path.join(‘D’, ‘Medium’, ‘New Folder’)will return
‘D\\Medium\\New Folder’
os.makedirs()— create a directory;
shutil.copy2(“sourcePath”, “destinationPath”)orshutil.move(“sourcePath”, “destinationPath”)— copy or cut a file respectively.
3. One-liner: Nested list comprehension to get rid of the for loops
Scenario: this “simple” task we come across is to combine several lists into one big long list,
Surely, we can write five nestedforloops to append each sublist to the final output list. But it’s smarter to turn to nested list comprehension for the most concise way,
Scenario: while Python is recognized as one of the most effective programming languages, data scientists still need to check the runtime of our programs.
It’s not the hardest thing if we just implement a bare-bones Python timer for each function we want to monitor. However, if we code it as a decorator, we would make our timer much easier to be version-controlled and reused!
Here is how,
In this snippet,
the timer is wrapped in atimerWrapperfunction, which then is used as a decorator called prior to the main function;
The example main function is to return a request connecting to an URL, which isone of my previous blogs.
Running this code gives us the time elapsed,
Current function: getArtile
Run Time: 1.6542516000008618Out[101]: <Response [200]>
Now, to time another function, all we need is to put the@timeWrapperin front of the function,
getMultiplication(3)
10
10000000000
10000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
Current function: getMultiplication
Run Time: 0.00014700000065204222
5. Leverage theoptionssystem to customize your display
Scenario: as data scientists, we analyze data with {pandas} and {numpy} on a daily basis. When I first learned coding in Python, I was frustrated seeing this after reading my data into the IDE,
Clearly, data display is cut off both row-wise and column-wise, and the following code can fix it,
Here, we are explicitly setting the maximum columns, rows and column width to display/print in the console. There are numerous customizableoptions and settingsin {pandas}, and similar operations are also available in{numpy}for arrays and matrix,
6. Reproduce your machine learning model results? Set seeds!
Scenario: Due to the stochastic nature of machine learning modeling process, we have all encountered the non-deterministic aspect of machine learning. This randomness results in our difficulty reproducing the same results across different runs. Consequently, it’s challenging to figure out whether an improvement in performance metrics is a result of successful model tuning or simply a different random training/testing sample.
Luckily the reproducibility can be achieved bysetting the random seedthroughout your model pipeline, provided that you do it correctly! How many times have you seen questions like “Getting a different result despite random seed defined” popping up on Stack Overflow? Well, how to appropriately set seeds should be in the first page of documentations, but it took me some time to dig it out.
I found that NOT every seed is defined the same in {numpy}, {sklearn}, {tensorflow} or {torch}. Thus, it’s a best practice to use a definitive function that sets all SEEDS for all your frameworks. For example,
Adding this tacticalreset_random_seed()function to all necessary steps of your workflow, such as train-test split, model compile/training, and interpretation, will get you half way to full reproducibility. More detailed visibility into your experiments will finish the second half!
새로운 아이디어는 어디에서 나오는가? 답은 간단하다. 바로 다름이다. 다양한 창의력 이론이 존재하지만, 이들 모두가 공유하는 유일한 교리는 ‘전혀 상관없어 보이는 것을 나란히 놓는 구조에서 창의성이 나온다’는 것이다. 다름을 최대화하는 최적의 방식은 연령과 문화, 학문을 섞는 것이다. - 니콜라스 네그로폰테, MIT 미디어랩 소장
다름을 인정하는 데서 창조가 시작됩니다. 다양성은 발전을 자극하고, 동질성은 발전을 늦춥니다. 다른 시각을 가지고 여러 가지 다른 이론들을 이리저리 배열해보면 새로운 아이디어가 떠오르기 마련입니다.
그러므로 쉰다는 것은 외부의 강제성을 벗어난 상태, 스스로를 지킬 수 있는 자기존엄성과 연결된다. 자기존엄성이란 자율적으로 결정할 수 있는 상태, 즉 '자기결정권'이 보장된 상태에서 가능하다. 이 자기결정권은 그저 단순한 의지의 표현이 아니라, 자기한테 필요한 자원에 접근할 수 있고 그 자원을 사용할 수 있는 권리를 포함한다.
- 이승원의《우리는 왜 쉬지 못하는가》중에서 -
* 쉼은 '내어맡김'입니다. '애씀'을 내려놓고, 내어맡길 때 진정한 쉼이 찾아옵니다. 그래서 욕조에서 긴장을 풀고 있을 때나, 단잠에서 깨어날 무렵에 섬광 같은 직감이 다가오기도 합니다. 아르키메데스의 '유레카'의 외침도 그렇게 생긴 것입니다. 쉼이 가져다주는 선물입니다.
