Skip to content
This repository was archived by the owner on Sep 12, 2024. It is now read-only.

Python3 Migration Notes

Amanda Goldani edited this page Aug 11, 2021 · 9 revisions

This page is intended to show the differences between the old version of the code and the new one.

Note that all Python3 top levels are located in src/python/, this was omitted in the details for convenience.


Cache

Includes all cache functions and classes from utils.py.

Details
Python 2 Python 3 tested notes
utils.py/cacheInfo.from_file Cache/CacheManager.py/CacheManager._getFromFile * called in get
utils.py/cacheInfo._file_key Cache/CacheManager.py/CacheManager._getKeyFilePath * called in get
utils.py/cacheInfo.get Cache/CacheManager.py/CacheManager.get
utils.py/cacheInfo.purge Cache/CacheManager.py/CacheManager.purge
utils.py/cacheInfo.store Cache/CacheManager.py/CacheManager.set
utils.py/docCache.get Cache/DataCacheLoader.py/DataCacheLoader.get
utils.py/docCache.__init__.cache Cache/DocumentCache.py * each possible document cache key became a class with the corresponding get method. The DataCacheLoader makes the mapping and loads the document data when required
Cache/CacheManager.py/CacheManager._buildMongoDocument * called in set
Cache/CacheManager.py/CacheManager._setMongoCollection * required by the MongoClient

Databases

Includes the helper function from utils.py that creates the connection to MongoDB, as well as the migration of the file that creates the connection to OracleDB.

Details
Python 2 Python 3 tested notes
utils.py/mongo_client Databases/Mongo/MongoClient.py * this is an abstract class where the mongo collection must be defined in the child classes
Unified/assignSession.py Databases/Oracle/OracleClient.py * tested through the OracleDB test file
Unified/assignSchema.py Databases/Oracle/OracleDB.py

MongoControllers

Includes all the helper classes from utils.py controlling a MongoDB collection.

AgentController:

Details
Python 2 Python 3 tested notes
utils.py/agentInfoDB MongoControllers/AgentController.py/AgentController._setMongoCollection * required by the MongoClient
utils.py/agentInfo.agentStatus MongoControllers/AgentController.py/AgentController.get * this will return all the agent's data, use result.get("status") to keep the same behavior
utils.py/agentInfo.speed_draining MongoControllers/AgentController.py/AgentController.getAgents(speeddrain=True)
utils.py/agentInfo.poll MongoControllers/AgentController.py/AgentController.poll
utils.py/agentInfo.change_status MongoControllers/AgentController.py/AgentController.set
utils.py/agentInfo.getStatus MongoControllers/AgentController.py/AgentController.syncToProduction
utils.py/agentInfo.checkTrello MongoControllers/AgentController.py/AgentController.syncToTrello
MongoControllers/AgentController.py/AgentController.updateTrelloBoard
* split because the Trello board was updated or the collection was sync to the board (never both!)
MongoControllers/AgentController.py/AgentController._areRunningOverThreshold * called in poll
MongoControllers/AgentController.py/AgentController._bootAgent * called in poll
MongoControllers/AgentController.py/AgentController._buildMongoDocument * called in set
MongoControllers/AgentController.py/AgentController._countAgentsWithReleaseStatus * called in poll
MongoControllers/AgentController.py/AgentController._countJobsAndCpus * called in poll
MongoControllers/AgentController.py/AgentController._drainAgent * called in poll
MongoControllers/AgentController.py/AgentController._initializePollParams * called in poll
MongoControllers/AgentController.py/AgentController._isLight * called in poll
MongoControllers/AgentController.py/AgentController._isStuffed * called in poll
MongoControllers/AgentController.py/AgentController._newRelease * called in poll
MongoControllers/AgentController.py/AgentController._retireAgent * called in poll
MongoControllers/AgentController.py/AgentController._setCandidatesForDraining * called in poll
MongoControllers/AgentController.py/AgentController._setCandidatesForOpenDraining * called in poll
MongoControllers/AgentController.py/AgentController._setCandidatesForSpeedDraining * called in poll
MongoControllers/AgentController.py/AgentController._setCandidatesForWakingUp * called in poll
MongoControllers/AgentController.py/AgentController._setLastestAndOldestReleases * called in poll
MongoControllers/AgentController.py/AgentController._setRecentAgents * called in poll
MongoControllers/AgentController.py/AgentController._setSpeedDrainingPriority * called in poll
MongoControllers/AgentController.py/AgentController._sumRunningCapacity * called in poll
MongoControllers/AgentController.py/AgentController._syncAgentStatus * called in syncToProduction
MongoControllers/AgentController.py/AgentController._tell * called in poll
MongoControllers/AgentController.py/AgentController._updateAgentConfig * called in poll
MongoControllers/AgentController.py/AgentController.getAgents

BatchController

Details
Python 2 Python 3 tested notes
utils.py/batchInfo.content MongoControllers/BatchController.py/BatchController.get
utils.py/batchInfo.all MongoControllers/BatchController.py/BatchController.getBatches
utils.py/batchInfo.pop MongoControllers/BatchController.py/BatchController.pop
utils.py/batchInfo.update
utils.py/batchInfo.add
MongoControllers/BatchController.py/BatchController.set * unified because add is only called inside update
MongoControllers/BatchController.py/BatchController._buildMongoDocument * called in set
MongoControllers/BatchController.py/BatchController._setMongoCollection * required by the MongoClient

CampaignController

Details
Python 2 Python 3 tested notes
utils.py/campaignInfo.content MongoControllers/CampaignController.py/CampaignController.get
utils.py/campaignInfo.all MongoControllers/CampaignController.py/CampaignController.getCampaigns
utils.py/campaignInfo.parameters MongoControllers/CampaignController.py/CampaignController.getCampaignParameters
utils.py/campaignInfo.allSecondaries MongoControllers/CampaignController.py/CampaignController.getSecondaries
utils.py/campaignInfo.get MongoControllers/CampaignController.py/CampaignController.getCampaignValue
utils.py/campaignInfo.go MongoControllers/CampaignController.py/CampaignController.go
utils.py/campaignInfo.pop MongoControllers/CampaignController.py/CampaignController.pop
utils.py/campaignInfo.update
utils.py/campaignInfo.add
MongoControllers/CampaignController.py/CampaignController.set * unified because add is only called inside update
MongoControllers/CampaignController.py/CampaignController._buildMongoDocument * called in set
MongoControllers/CampaignController.py/CampaignController._setCampaigns * called when instantiated
MongoControllers/CampaignController.py/CampaignController._setMongoCollection * required by the MongoClient

CloseoutController

Details
Python 2 Python 3 tested notes
utils.py/closeoutInfo.one_line MongoControllers/CloseoutController.py/CloseoutController._buildHtmlRow * called in assistance and summary
utils.py/closeoutInfo.assistance MongoControllers/CloseoutController.py/CloseoutController.assistance
utils.py/closeoutInfo.pop MongoControllers/CloseoutController.py/CloseoutController.clean
utils.py/closeoutInfo.get MongoControllers/CloseoutController.py/CloseoutController.get
utils.py/closeoutInfo.html MongoControllers/CloseoutController.py/CloseoutController.html
utils.py/closeoutInfo.update MongoControllers/CloseoutController.py/CloseoutController.set
utils.py/closeoutInfo.summary MongoControllers/CloseoutController.py/CloseoutController.summary
utils.py/closeoutInfo.table_header Included in the CloseoutController templates
MongoControllers/CloseoutController.py/CloseoutController._buildMongoDocument * called in set
MongoControllers/CloseoutController.py/CloseoutController._renderAssistanceHtml * called in assistance
MongoControllers/CloseoutController.py/CloseoutController._renderSummaryHtml * called in summary
MongoControllers/CloseoutController.py/CloseoutController._setMongoCollection * required by the MongoClient
MongoControllers/CloseoutController.py/CloseoutController.getWorkflows

ModuleLockController

Details
Python 2 Python 3 tested notes
utils.py/moduleLock.check MongoControllers/ModuleLockController.py/ModuleLockController.check
utils.py/moduleLock.clean MongoControllers/ModuleLockController.py/ModuleLockController.clean
utils.py/moduleLock.all_locks MongoControllers/ModuleLockController.py/ModuleLockController.get
MongoControllers/ModuleLockController.py/ModuleLockController._buildMongoDocument * called in set
MongoControllers/ModuleLockController.py/ModuleLockController._setMongoCollection * required by the MongoClient
MongoControllers/ModuleLockController.py/ModuleLockController.go
MongoControllers/ModuleLockController.py/ModuleLockController.pop
MongoControllers/ModuleLockController.py/ModuleLockController.set

RemainingDatasetController

Details
Python 2 Python 3 tested notes
utils.py/remainingDatasetInfo.clean MongoControllers/RemainingDatasetController.py/RemainingDatasetController.clean
utils.py/remainingDatasetInfo.get MongoControllers/RemainingDatasetController.py/RemainingDatasetController.get
utils.py/remainingDatasetInfo.sites MongoControllers/RemainingDatasetController.py/RemainingDatasetController.getSites
utils.py/remainingDatasetInfo.purge MongoControllers/RemainingDatasetController.py/RemainingDatasetController.purge
utils.py/remainingDatasetInfo.set MongoControllers/RemainingDatasetController.py/RemainingDatasetController.set
utils.py/remainingDatasetInfo.sync MongoControllers/RemainingDatasetController.py/RemainingDatasetController.sync
MongoControllers/RemainingDatasetController.py/RemainingDatasetController.syncSite
* split to let explicit if one site or all sites are being sync
utils.py/remainingDatasetInfo.tell MongoControllers/RemainingDatasetController.py/RemainingDatasetController.tell
MongoControllers/RemainingDatasetController.py/RemainingDatasetController._buildMongoDocument * called in set
MongoControllers/RemainingDatasetController.py/RemainingDatasetController._setMongoCollection * required by the MongoClient

ReportController

Details
Python 2 Python 3 tested notes
utils.py/reportInfo._convert MongoControllers/ReportController.py/ReportController._convertValues * called in set
utils.py/reportInfo.get MongoControllers/ReportController .py/ReportController.get
utils.py/reportInfo.purge MongoControllers/ReportController .py/ReportController.purge
MongoControllers/ReportController .py/ReportController.clean
MongoControllers/ReportController .py/ReportController.cleanWorkflow
* split because the old function would do different things depending os the input params
utils.py/reportInfo._put MongoControllers/ReportController.py/ReportController.set
utils.py/reportInfo.set_blocks MongoControllers/ReportController.py/ReportController.setWorkflowBlocks
utils.py/reportInfo.set_errors MongoControllers/ReportController.py/ReportController.setWorkflowErrors
utils.py/reportInfo.set_files MongoControllers/ReportController.py/ReportController.setWorkflowFiles
utils.py/reportInfo.set_IO MongoControllers/ReportController.py/ReportController.setWorkflowIO
utils.py/reportInfo.set_logs MongoControllers/ReportController.py/ReportController.setWorkflowLogs
utils.py/reportInfo.set_missing MongoControllers/ReportController.py/ReportController.setWorkflowMissing
utils.py/reportInfo._set_for_task MongoControllers/ReportController.py/ReportController.setWorkflowTask
utils.py/reportInfo.set_ufiles MongoControllers/ReportController.py/ReportController.setWorkflowUFiles
MongoControllers/ReportController.py/ReportController._buildMongoDocument * called in set
MongoControllers/ReportController.py/ReportController._setMongoCollection * required by the MongoClient

StartStopTimeController

Details
Python 2 Python 3 tested notes
utils.py/StartStopInfo.get MongoControllers/StartStopTimeController.py/StartStopTimeController.get
utils.py/StartStopInfo.purge MongoControllers/StartStopTimeController.py/StartStopTimeController.purge
utils.py/StartStopInfo.pushStartStopTime MongoControllers/StartStopTimeController.py/StartStopTimeController.set
MongoControllers/StartStopTimeController.py/StartStopTimeController._buildMongoDocument * called in set
MongoControllers/StartStopTimeController.py/StartStopTimeController._setMongoCollection * required by the MongoClient

StatusHistoryController

Details
Python 2 Python 3 tested notes
utils.py/statusHistory.py/statusHistory.content MongoControllers/StatusHistoryController.py/StatusHistoryController.get
utils.py/statusHistory.py/statusHistory.trim MongoControllers/StatusHistoryController.py/StatusHistoryController.purge
utils.py/statusHistory.py/statusHistory.add MongoControllers/StatusHistoryController.py/StatusHistoryController.set
MongoControllers/StatusHistoryController.py/StatusHistoryController._buildMongoDocument * called in set
MongoControllers/StatusHistoryController.py/StatusHistoryController._setMongoCollection * required by the MongoClient

WTCController

Details
Python 2 Python 3 tested notes
utils.py/wtcInfo.clean MongoControllers/WTCController.py/WTCController.clean
utils.py/wtcInfo.remove MongoControllers/WTCController.py/WTCController.cleanKeyword
utils.py/wtcInfo._get MongoControllers/WTCController.py/WTCController.get
utils.py/wtcInfo.getBypass MongoControllers/WTCController.py/WTCController.getBypass
utils.py/wtcInfo.getForce MongoControllers/WTCController.py/WTCController.getForce
utils.py/wtcInfo.getHold MongoControllers/WTCController.py/WTCController.getHold
utils.py/wtcInfo.add MongoControllers/WTCController.py/WTCController.set
MongoControllers/WTCController.py/WTCController._buildMongoDocument * called in set
MongoControllers/WTCController.py/WTCController._setMongoCollection * required by the MongoClient

Services

Includes all helper functions from utils.py talking to a service.

DBS

Details
Python 2 Python 3 tested notes
utils.py/componentCheck.check_dbs Services/DBS/DBSReader.py/DBSReader.check
utils.py/getFileBlock Services/DBS/DBSReader.py/DBSReader.getBlockName
utils.py/getDatasetBlocks Services/DBS/DBSReader.py/DBSReader.getDatasetBlockNames
Services/DBS/DBSReader.py/DBSReader.getDatasetBlockNamesByLumis
Services/DBS/DBSReader.py/DBSReader.getDatasetBlockNamesByRuns
* split because the block names were filtered by runs or by lumis (never both!)
utils.py/getDatasetEventsAndLumis Services/DBS/DBSReader.py/DBSReader.getDatasetEventsAndLumis
Services/DBS/DBSReader.py/DBSReader.getBlocksEventsAndLumis
* split because events and lumis were got by dataset or blocks (never both!)
utils.py/getDatasetFileArray Services/DBS/DBSReader.py/DBSReader.getDatasetFiles * input args use_array and cache_timeout were removed because were not used
utils.py/getDatasetLumisAndFiles Services/DBS/DBSReader.py/DBSReader.getDatasetLumisAndFiles * only does the request, input args runs and lumilist were removed but the helper functions Utilities/DataTools.py/filterLumisAndFilesByRuns or filterLumisAndFilesByLumis can be used if needed
utils.py/getDatasets Services/DBS/DBSReader.py/DBSReader.getDatasetNames
utils.py/findParent Services/DBS/DBSReader.py/DBSReader.getDatasetParent
utils.py/getDatasetRuns Services/DBS/DBSReader.py/DBSReader.getDatasetRuns
utils.py/getDatasetSize Services/DBS/DBSReader.py/DBSReader.getDatasetSize
utils.py/getDatasetStatus Services/DBS/DBSReader.py/DBSReader.getDBSStatus
utils.py/getFilesWithLumiInRun Services/DBS/DBSReader.py/DBSReader.getFilesWithLumiInRun
utils.py/getLFNbase Services/DBS/DBSReader.py/DBSReader.getLFNBase
utils.py/setDatasetStatus Services/DBS/DBSWriter.py/DBSWriter.setDatasetStatus
utils.py/setFileStatus Services/DBS/DBSWriter.py/DBSWriter.setFileStatus
Services/DBS/DBSReader.py/DBSReader._getBlockFileLumis * called in getBlocksLumisAndFilesForCaching
Services/DBS/DBSReader.py/DBSReader._getFileLumiArray * called in getFilesWithLumiInRun
Services/DBS/DBSReader.py/DBSReader.getBlocksLumisAndFilesForCaching
Services/DBS/DBSReader.py/DBSReader.getRecoveryBlocksAndLocations

EOS

Details
Python 2 Python 3 tested notes
utils.py/eosRead Services/EOS/EOSReader.read
utils.py/eosFile.close Services/EOS/EOSWriter.save
utils.py/eosFile.write Services/EOS/EOSWriter.write

ReqMgr

Details
Python 2 Python 3 tested notes
utils.py/getAgentConfig Services/ReqMgr/ReqMgrReader.py/ReqMgrReader.getAgentConfig
utils.py/getReqmgrInfo Services/ReqMgr/ReqMgrReader.py/ReqMgrReader.getReqmgrInfo
utils.py/getWorkflowByCampaign Services/ReqMgr/ReqMgrReader.py/ReqMgrReader.getWorkflowsByCampaign
utils.py/getWorkflowByName Services/ReqMgr/ReqMgrReader.py/ReqMgrReader.getWorkflowsByNames
utils.py/getWorkflowByOutput Services/ReqMgr/ReqMgrReader.py/ReqMgrReader.getWorkflowsByOutput * endpoint changed from CouchDB to the RESTful api
utils.py/getWorkflowById Services/ReqMgr/ReqMgrReader.py/ReqMgrReader.getWorkflowsByPrepId * endpoint changed from CouchDB to the RESTful api
utils.py/getWorkflows Services/ReqMgr/ReqMgrReader.py/ReqMgrReader.getWorkflowsByStatus
utils.py/getWorkload Services/ReqMgr/ReqMgrReader.py/ReqMgrReader.getWorkflowSchema
utils.py/workflowInfo.getSchema Services/ReqMgr/ReqMgrReader.py/ReqMgrReader.getWorkflowSchema * use makeCopy = True to keep the same behavior
utils.py/workflowInfo.getSplittingsNew Services/ReqMgr/ReqMgrReader.py/ReqMgrReader.getSplittingsSchema * only does the request, input args strip and allTasks were removed but the helper functions Utilities/DataTools.py/filterSplittingsParam and filterSplittingsTaskTypes can be used if needed
utils.py/sendAgentConfig Services/ReqMgr/ReqMgrWriter.py/ReqMgrWriter.setAgentConfig
Services/ReqMgr/ReqMgrReader.py/ReqMgrReader.getWorkflowsByParam

Trello

Details
Python 2 Python 3 tested notes
TrelloClient.py/_get Services/Trello/TrelloClient.py/TrelloClient._getResponse * called in getBoard and getCard
TrelloClient.py/_put Services/Trello/TrelloClient.py/TrelloClient._set * called in setList
TrelloClient.py/getAgents Services/Trello/TrelloClient.py/TrelloClient._syncAgents * called when instantiated
TrelloClient.py/getBoard Services/Trello/TrelloClient.py/TrelloClient.getBoard
TrelloClient.py/getCard Services/Trello/TrelloClient.py/TrelloClient.getCard
TrelloClient.py/changeList Services/Trello/TrelloClient.py/TrelloClient.setList
Services/Trello/TrelloClient.py/TrelloClient.getListId

WMStatsReader

Details
Python 2 Python 3 tested notes
utils.py/getAllAgents Services/WMStats/WMStatsReader.py/WMStatsReader.getAgents
Services/WMStats/WMStatsReader.py/WMStatsReader.getProductionAgents

Utilities

Includes all helper functions from utils.py that are used for authentication, configuration, etc.

Details
Python 2 Python 3 tested notes
utils.py/make_x509_conn Utilities/Authenticate.py/getX509Conn
utils.py/unifiedConfiguration Utilities/ConfigurationHandler.py/ConfigurationHandler
utils.py/ThreadHandler Utilities/Decorators.py/runWithMultiThreading
utils.py/runWithRetries Utilities/Decorators.py/runWithRetries
utils.py/display_N Utilities/Logging/displayNumber
utils.py/display_time Utilities/Logging/displayTime
utils.py/GET Utilities/WebTools.py/getResponse
Utilities/DataTools/filterFilesAndLocationsInDBS
Utilities/DataTools/filterLumisAndFilesByLumis
Utilities/DataTools/filterLumisAndFilesByRuns
Utilities/DataTools/filterRecoveryFilesAndLocations
Utilities/DataTools/filterSplittingsParam
Utilities/DataTools/filterSplittingsTaskTypes
Utilities/DataTools/sortByWakeUpPriority
Utilities/IteratorTools/filterKeys
Utilities/IteratorTools/mapKeys
Utilities/IteratorTools/mapValues
Utilities/WebTools.py/sendResponse

Clone this wiki locally